[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:04:05 dist-test-slave-nsrc krb5kdc[23479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:04:05 dist-test-slave-nsrc krb5kdc[23479](info): set up 2 sockets
Sep 25 15:04:05 dist-test-slave-nsrc krb5kdc[23479](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 15:04:06.333220 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.002s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:04:06 dist-test-slave-nsrc krb5kdc[23479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:04:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:04:06.487751 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:36563
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:36563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:06.801748 23492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:06.802021 23492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:06.802109 23492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:06.809826 23492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:06.809950 23492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:06.810001 23492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:06.810053 23492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:06.810096 23492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:04:06.819772 23492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:36563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:36563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:06.821085 23492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:06.822347 23492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:06.833770 23497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:06.833841 23500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:06.833798 23498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:06.834522 23492 server_base.cc:1034] running on GCE node
I20260925 15:04:06.835278 23492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:06.836688 23492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:06.837885 23492 hybrid_clock.cc:648] HybridClock initialized: now 1790348646837837 us; error 39 us; skew 500 ppm
I20260925 15:04:06.838203 23492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:06 dist-test-slave-nsrc krb5kdc[23479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348646, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:06.845005 23492 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:04:06.846428 23492 webserver.cc:460] Webserver started at http://127.22.236.62:36363/ using document root <none> and password file <none>
I20260925 15:04:06.847077 23492 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:06.847170 23492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:06.847433 23492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:06.850390 23492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "9961f821e27e43d1bc4568d3b0917c90"
format_stamp: "Formatted at 2026-09-25 15:04:06 on dist-test-slave-nsrc"
server_key: "6043b4ec83b729bd048ab2389c90e1c1"
server_key_iv: "bf29390640b9bde0f07a48e3b5212fc6"
server_key_version: "encryptionkey@0"
I20260925 15:04:06.851480 23492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "9961f821e27e43d1bc4568d3b0917c90"
format_stamp: "Formatted at 2026-09-25 15:04:06 on dist-test-slave-nsrc"
server_key: "6043b4ec83b729bd048ab2389c90e1c1"
server_key_iv: "bf29390640b9bde0f07a48e3b5212fc6"
server_key_version: "encryptionkey@0"
I20260925 15:04:06.856657 23492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 15:04:06.860074 23507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:06.861477 23492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:04:06.861651 23492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "9961f821e27e43d1bc4568d3b0917c90"
format_stamp: "Formatted at 2026-09-25 15:04:06 on dist-test-slave-nsrc"
server_key: "6043b4ec83b729bd048ab2389c90e1c1"
server_key_iv: "bf29390640b9bde0f07a48e3b5212fc6"
server_key_version: "encryptionkey@0"
I20260925 15:04:06.861828 23492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:06.882849 23492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:06.886682 23492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:06.886943 23492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:06.904578 23492 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:36563
I20260925 15:04:06.904596 23558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:36563 every 8 connection(s)
I20260925 15:04:06.905851 23492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:06.909747 23559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:06.913255 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 23492
I20260925 15:04:06.913527 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:06.915271 23472 external_mini_cluster.cc:1428] Setting key 4a699ec6a99d03972ea09812b6bacbeb
I20260925 15:04:06.919972 23559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90: Bootstrap starting.
I20260925 15:04:06.922943 23559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:06.923861 23559 log.cc:824] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:06.926265 23559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90: No bootstrap required, opened a new log
I20260925 15:04:06.934623 23559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9961f821e27e43d1bc4568d3b0917c90" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36563 } }
I20260925 15:04:06.934945 23559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:06.935047 23559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9961f821e27e43d1bc4568d3b0917c90, State: Initialized, Role: FOLLOWER
I20260925 15:04:06.935626 23559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9961f821e27e43d1bc4568d3b0917c90" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36563 } }
I20260925 15:04:06.935835 23559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:06.935957 23559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:06.936102 23559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:06.938148 23559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9961f821e27e43d1bc4568d3b0917c90" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36563 } }
I20260925 15:04:06.938621 23559 leader_election.cc:302] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9961f821e27e43d1bc4568d3b0917c90; no voters:
I20260925 15:04:06.939195 23559 leader_election.cc:288] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:06.939303 23564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:06.940819 23564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [term 1 LEADER]: Becoming Leader. State: Replica: 9961f821e27e43d1bc4568d3b0917c90, State: Running, Role: LEADER
I20260925 15:04:06.941385 23564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9961f821e27e43d1bc4568d3b0917c90" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36563 } }
I20260925 15:04:06.941908 23559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 15:04:06 dist-test-slave-nsrc krb5kdc[23479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:06.950552 23566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9961f821e27e43d1bc4568d3b0917c90. Latest consensus state: current_term: 1 leader_uuid: "9961f821e27e43d1bc4568d3b0917c90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9961f821e27e43d1bc4568d3b0917c90" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36563 } } }
I20260925 15:04:06.950922 23566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:06.951404 23565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9961f821e27e43d1bc4568d3b0917c90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9961f821e27e43d1bc4568d3b0917c90" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36563 } } }
I20260925 15:04:06.955582 23565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:06.957436 23575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:06.958801 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:06.919021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.22.236.62:36563)
0925 15:04:06.919734 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:06.919745 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:06.920991 (+ 1246us) server_negotiation.cc:407] Connection header received
0925 15:04:06.930622 (+ 9631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:06.930642 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:06.931020 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:06.931339 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:06.934034 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:06.935344 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:06.937135 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:06.937808 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:06.937980 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:06.944009 (+ 6029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:06.944020 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:06.944040 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:06.944086 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:06.947138 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:06.947967 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:06.947974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:06.947980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:06.948170 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:06.948696 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:06.948701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:06.948705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:06.949232 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:06.955908 (+ 6676us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:06.956417 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 15:04:06.956999 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":337,"threads_started":1}
I20260925 15:04:06.962333 23575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:06.971211 23575 catalog_manager.cc:1347] Generated new cluster ID: 6c1e7948eb6f4dd89f571c3bfa319de9
I20260925 15:04:06.971323 23575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:06.985159 23575 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:06.986409 23575 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:06.995428 23575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9961f821e27e43d1bc4568d3b0917c90: Generated new TSK 0
I20260925 15:04:06.996295 23575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:04:07.039242 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36563
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:07.371958 23585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:07.372263 23585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:07.372411 23585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:07.379499 23585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:07.379602 23585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:07.379791 23585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:04:07.389745 23585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36563
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:07.391188 23585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:07.392632 23585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:07.408430 23590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:07.412508 23591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:07.412750 23593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:07.413710 23585 server_base.cc:1034] running on GCE node
I20260925 15:04:07.414315 23585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:07.415247 23585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:07.416436 23585 hybrid_clock.cc:648] HybridClock initialized: now 1790348647416385 us; error 42 us; skew 500 ppm
I20260925 15:04:07.416729 23585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:07 dist-test-slave-nsrc krb5kdc[23479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348647, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:07.419597 23585 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:04:07.421090 23585 webserver.cc:460] Webserver started at http://127.22.236.1:39787/ using document root <none> and password file <none>
I20260925 15:04:07.421756 23585 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:07.421877 23585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:07.422168 23585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:07.425102 23585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "edfe71fe1fb94aa182df4e4d0673150f"
format_stamp: "Formatted at 2026-09-25 15:04:07 on dist-test-slave-nsrc"
server_key: "c9f8473fa3a31510cc7540b56160cffb"
server_key_iv: "f2dd30e89706b2d43a9c2423fc5cf6c4"
server_key_version: "encryptionkey@0"
I20260925 15:04:07.425873 23585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "edfe71fe1fb94aa182df4e4d0673150f"
format_stamp: "Formatted at 2026-09-25 15:04:07 on dist-test-slave-nsrc"
server_key: "c9f8473fa3a31510cc7540b56160cffb"
server_key_iv: "f2dd30e89706b2d43a9c2423fc5cf6c4"
server_key_version: "encryptionkey@0"
I20260925 15:04:07.430740 23585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:04:07.434008 23600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:07.435249 23585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:04:07.435391 23585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "edfe71fe1fb94aa182df4e4d0673150f"
format_stamp: "Formatted at 2026-09-25 15:04:07 on dist-test-slave-nsrc"
server_key: "c9f8473fa3a31510cc7540b56160cffb"
server_key_iv: "f2dd30e89706b2d43a9c2423fc5cf6c4"
server_key_version: "encryptionkey@0"
I20260925 15:04:07.435563 23585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:07.449635 23585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:07.453330 23585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:07.453573 23585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:07.454460 23585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:07.455722 23585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:07.455826 23585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:07.455973 23585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:07.456039 23585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:07.485249 23585 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:38331
I20260925 15:04:07.485289 23712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:38331 every 8 connection(s)
I20260925 15:04:07.486440 23585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:04:07.495431 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 23585
I20260925 15:04:07.495632 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:04:07.496227 23472 external_mini_cluster.cc:1428] Setting key e3d26d1589893f3ae65f6a9f4b4ae5d1
Sep 25 15:04:07 dist-test-slave-nsrc krb5kdc[23479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348647, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:07.506145 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:07.489283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:35213 (local address 127.22.236.62:36563)
0925 15:04:07.489712 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:07.489717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:07.490589 (+ 872us) server_negotiation.cc:407] Connection header received
0925 15:04:07.492332 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:07.492338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:07.492426 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:07.492522 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:07.494016 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:07.494752 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:07.496539 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:07.497013 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:07.497122 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:07.500699 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:07.500704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:07.500708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:07.500739 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:07.503075 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:07.503761 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:07.503765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:07.503767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:07.503907 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:07.504368 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:07.504371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:07.504373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:07.504624 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:07.504802 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:07.505593 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 15:04:07.505803 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":223,"threads_started":1}
I20260925 15:04:07.507027 23716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:07.489750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36563 (local address 127.22.236.1:35213)
0925 15:04:07.490409 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:07.490468 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:07.492039 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:07.492777 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:07.492785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:07.493287 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:07.493848 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:07.493865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:07.494899 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:07.494904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:07.496404 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:07.496416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:07.497218 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:07.497225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:07.497537 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:07.498204 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:07.498226 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:07.500486 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:07.503218 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:07.503227 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:07.503245 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:07.503636 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:07.504013 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:07.504018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:07.504021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:07.504253 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:07.504758 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:07.504767 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:07.505115 (+ 348us) client_negotiation.cc:769] Sending connection context
0925 15:04:07.505340 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 15:04:07.505860 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":250,"threads_started":1}
I20260925 15:04:07.508778 23713 heartbeater.cc:347] Connected to a master server at 127.22.236.62:36563
I20260925 15:04:07.509132 23713 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
I20260925 15:04:07.509856 23713 heartbeater.cc:511] Master 127.22.236.62:36563 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
I20260925 15:04:07.511929 23524 ts_manager.cc:194] Registered new tserver with Master: edfe71fe1fb94aa182df4e4d0673150f (127.22.236.1:38331)
I20260925 15:04:07.513409 23524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:35213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:04:07.530522 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36563
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:07.841507 23719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:07.841786 23719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:07.841924 23719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:07.848884 23719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:07.848990 23719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:07.849136 23719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:04:07.858510 23719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36563
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:07.859710 23719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:07.860967 23719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:07.873106 23727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:07.873158 23724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:07.873162 23725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:07.874006 23719 server_base.cc:1034] running on GCE node
I20260925 15:04:07.874574 23719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:07.875413 23719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:07.876637 23719 hybrid_clock.cc:648] HybridClock initialized: now 1790348647876598 us; error 28 us; skew 500 ppm
I20260925 15:04:07.877313 23719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:07 dist-test-slave-nsrc krb5kdc[23479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348647, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:07.880086 23719 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:04:07.881534 23719 webserver.cc:460] Webserver started at http://127.22.236.2:36177/ using document root <none> and password file <none>
I20260925 15:04:07.882159 23719 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:07.882268 23719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:07.882542 23719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:07.885339 23719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "4089da902de54fa1a2e525b7788d98f0"
format_stamp: "Formatted at 2026-09-25 15:04:07 on dist-test-slave-nsrc"
server_key: "f1ca1eb8941b022840068e3afa101d15"
server_key_iv: "0ce3d88fe8f23e10bf903d988c173846"
server_key_version: "encryptionkey@0"
I20260925 15:04:07.886049 23719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "4089da902de54fa1a2e525b7788d98f0"
format_stamp: "Formatted at 2026-09-25 15:04:07 on dist-test-slave-nsrc"
server_key: "f1ca1eb8941b022840068e3afa101d15"
server_key_iv: "0ce3d88fe8f23e10bf903d988c173846"
server_key_version: "encryptionkey@0"
I20260925 15:04:07.890949 23719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 15:04:07.894215 23734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:07.895565 23719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:04:07.895700 23719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "4089da902de54fa1a2e525b7788d98f0"
format_stamp: "Formatted at 2026-09-25 15:04:07 on dist-test-slave-nsrc"
server_key: "f1ca1eb8941b022840068e3afa101d15"
server_key_iv: "0ce3d88fe8f23e10bf903d988c173846"
server_key_version: "encryptionkey@0"
I20260925 15:04:07.895874 23719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:07.924965 23719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:07.928580 23719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:07.928833 23719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:07.929733 23719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:07.931349 23719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:07.931416 23719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:07.931519 23719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:07.931565 23719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:07.962275 23719 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:36509
I20260925 15:04:07.962320 23846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:36509 every 8 connection(s)
I20260925 15:04:07.963459 23719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:04:07.964182 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 23719
I20260925 15:04:07.964381 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:04:07.964897 23472 external_mini_cluster.cc:1428] Setting key dbe03492be3128026a2ca410d03a373f
Sep 25 15:04:07 dist-test-slave-nsrc krb5kdc[23479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348647, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
I20260925 15:04:07.982949 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:07.966260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:39601 (local address 127.22.236.62:36563)
0925 15:04:07.966446 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:07.966452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:07.967545 (+ 1093us) server_negotiation.cc:407] Connection header received
0925 15:04:07.969286 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:07.969291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:07.969369 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:07.969453 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:07.971303 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:07.972007 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:07.973548 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:07.973940 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:07.974044 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:07.977491 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:07.977496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:07.977499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:07.977528 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:07.979994 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:07.980743 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:07.980748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:07.980752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:07.980923 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:07.981323 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:07.981329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:07.981332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:07.981655 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:07.981824 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:07.982411 (+ 587us) server_negotiation.cc:299] Negotiation successful
0925 15:04:07.982623 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:04:07.983693 23849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:07.966727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36563 (local address 127.22.236.2:39601)
0925 15:04:07.967398 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:07.967434 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:07.969030 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:07.969689 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:07.969697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:07.970188 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:07.971125 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:07.971144 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:07.972181 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:07.972187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:07.973420 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:07.973431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:07.974139 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:07.974146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:07.974427 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:07.975007 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:07.975027 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:07.977277 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:07.980111 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:07.980118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:07.980130 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:07.980617 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:07.981034 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:07.981037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:07.981040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:07.981209 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:07.981766 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:07.981771 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:07.982071 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 15:04:07.982273 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 15:04:07.982628 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":267,"threads_started":1}
I20260925 15:04:07.985080 23847 heartbeater.cc:347] Connected to a master server at 127.22.236.62:36563
I20260925 15:04:07.985379 23847 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:07.986040 23847 heartbeater.cc:511] Master 127.22.236.62:36563 requested a full tablet report, sending...
I20260925 15:04:07.987521 23524 ts_manager.cc:194] Registered new tserver with Master: 4089da902de54fa1a2e525b7788d98f0 (127.22.236.2:36509)
I20260925 15:04:07.988607 23524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:39601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:04:08.000650 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36563
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:08.306458 23852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:08.306735 23852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:08.306857 23852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:08.314141 23852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:08.314249 23852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:08.314397 23852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:04:08.323565 23852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36563
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:08.324831 23852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:08.326077 23852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:08.338352 23857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:08.338400 23858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:08.338425 23860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:08.339251 23852 server_base.cc:1034] running on GCE node
I20260925 15:04:08.339798 23852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:08.340700 23852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:08.341957 23852 hybrid_clock.cc:648] HybridClock initialized: now 1790348648341886 us; error 46 us; skew 500 ppm
I20260925 15:04:08.342252 23852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:08 dist-test-slave-nsrc krb5kdc[23479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348648, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:08.345129 23852 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:04:08.346491 23852 webserver.cc:460] Webserver started at http://127.22.236.3:41079/ using document root <none> and password file <none>
I20260925 15:04:08.347108 23852 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:08.347216 23852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:08.347488 23852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:08.350692 23852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "b853948e69b348c5b0df8c3cd355d6d6"
format_stamp: "Formatted at 2026-09-25 15:04:08 on dist-test-slave-nsrc"
server_key: "877961e072c3fc3669535634c11f2b13"
server_key_iv: "cef3f4c8f3b13310ab184707ef180821"
server_key_version: "encryptionkey@0"
I20260925 15:04:08.351382 23852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "b853948e69b348c5b0df8c3cd355d6d6"
format_stamp: "Formatted at 2026-09-25 15:04:08 on dist-test-slave-nsrc"
server_key: "877961e072c3fc3669535634c11f2b13"
server_key_iv: "cef3f4c8f3b13310ab184707ef180821"
server_key_version: "encryptionkey@0"
I20260925 15:04:08.356112 23852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 15:04:08.359282 23867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:08.360563 23852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:08.360703 23852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "b853948e69b348c5b0df8c3cd355d6d6"
format_stamp: "Formatted at 2026-09-25 15:04:08 on dist-test-slave-nsrc"
server_key: "877961e072c3fc3669535634c11f2b13"
server_key_iv: "cef3f4c8f3b13310ab184707ef180821"
server_key_version: "encryptionkey@0"
I20260925 15:04:08.360896 23852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:08.374404 23852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:08.378154 23852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:08.378412 23852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:08.379700 23852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:08.380996 23852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:08.381069 23852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:08.381166 23852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:08.381217 23852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:08.413209 23852 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:34307
I20260925 15:04:08.413241 23979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:34307 every 8 connection(s)
I20260925 15:04:08.414402 23852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:04:08.423374 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 23852
I20260925 15:04:08.423547 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:04:08.423938 23472 external_mini_cluster.cc:1428] Setting key ad534bca58e9d61c43797c1eeb350139
Sep 25 15:04:08 dist-test-slave-nsrc krb5kdc[23479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348648, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:08.433413 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:08.416844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:53749 (local address 127.22.236.62:36563)
0925 15:04:08.416999 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:08.417003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:08.418149 (+ 1146us) server_negotiation.cc:407] Connection header received
0925 15:04:08.420317 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:08.420323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:08.420406 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:08.420493 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:08.421984 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:08.422722 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:08.424346 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:08.424795 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:08.424924 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:08.428395 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:08.428400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:08.428403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:08.428434 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:08.431020 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:08.431624 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:08.431628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:08.431630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:08.431733 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:08.432061 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:08.432063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:08.432065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:08.432354 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:08.432455 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:08.433018 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 15:04:08.433163 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:04:08.434391 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:08.417319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36563 (local address 127.22.236.3:53749)
0925 15:04:08.417991 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:08.418027 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:08.420007 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:08.420736 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:08.420745 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:08.421248 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:08.421803 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:08.421821 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:08.422871 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:08.422875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:08.424207 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:08.424219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:08.425030 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:08.425037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:08.425294 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:08.426008 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:08.426033 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:08.428200 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:08.431133 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:08.431141 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:08.431153 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:08.431535 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:08.431807 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:08.431810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:08.431812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:08.431980 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:08.432444 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:08.432450 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:08.432741 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 15:04:08.432951 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 15:04:08.433317 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":254,"threads_started":1}
I20260925 15:04:08.435649 23980 heartbeater.cc:347] Connected to a master server at 127.22.236.62:36563
I20260925 15:04:08.435927 23980 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:08.436640 23980 heartbeater.cc:511] Master 127.22.236.62:36563 requested a full tablet report, sending...
I20260925 15:04:08.437922 23524 ts_manager.cc:194] Registered new tserver with Master: b853948e69b348c5b0df8c3cd355d6d6 (127.22.236.3:34307)
I20260925 15:04:08.438936 23524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:53749
I20260925 15:04:08.439646 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:04:08 dist-test-slave-nsrc krb5kdc[23479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348648, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:04:08 dist-test-slave-nsrc krb5kdc[23479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348648, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:04:08.479202 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:08.465288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.22.236.1:38331)
0925 15:04:08.465845 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:08.465855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:08.465942 (+ 87us) server_negotiation.cc:407] Connection header received
0925 15:04:08.466081 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:08.466088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:08.466311 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:08.466441 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:08.467775 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:08.468864 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:08.470123 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:08.470644 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:08.470754 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:08.474391 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:08.474401 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:08.474416 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:08.474447 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:08.477017 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:08.477582 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:08.477588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:08.477590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:08.477696 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:08.477990 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:08.477993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:08.477994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:08.478292 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:08.478503 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:08.478736 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 15:04:08.478926 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":308,"threads_started":1}
W20260925 15:04:08.480468 23628 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37240
Sep 25 15:04:08 dist-test-slave-nsrc krb5kdc[23479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:04:08 dist-test-slave-nsrc krb5kdc[23479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:04:08.516403 23713 heartbeater.cc:503] Master 127.22.236.62:36563 was elected leader, sending a full tablet report...
I20260925 15:04:08.521844 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:08.508820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.22.236.1:38331)
0925 15:04:08.509009 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:08.509015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:08.509353 (+ 338us) server_negotiation.cc:407] Connection header received
0925 15:04:08.509531 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:08.509535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:08.509607 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:08.509689 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:08.510927 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:08.511600 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:08.512847 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:08.513231 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:08.513336 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:08.517106 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:08.517111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:08.517114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:08.517145 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:08.519670 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:08.520340 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:08.520345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:08.520347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:08.520465 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:08.520776 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:08.520780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:08.520782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:08.521028 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:08.521198 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:08.521424 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 15:04:08.521579 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:04:08.523236 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 23585
I20260925 15:04:08.534943 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:08.524085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.22.236.1:38331)
0925 15:04:08.524283 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:08.524289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:08.524558 (+ 269us) server_negotiation.cc:407] Connection header received
0925 15:04:08.524674 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:08.524678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:08.524729 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:08.524800 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:08.526327 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:08.526977 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:08.528120 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:08.528564 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:08.528653 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:08.530684 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:08.530688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:08.530691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:08.530713 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:08.532919 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:08.533492 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:08.533495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:08.533497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:08.533606 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:08.533859 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:08.533862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:08.533864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:08.534093 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:08.534246 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:08.534476 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 15:04:08.534622 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:04:08.535595 23708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:08.615756 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 23585
I20260925 15:04:08.632834 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 23719
Sep 25 15:04:08 dist-test-slave-nsrc krb5kdc[23479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:04:08.647322 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:08.633643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35688 (local address 127.22.236.2:36509)
0925 15:04:08.634131 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:08.634138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:08.634163 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:04:08.634221 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:08.634227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:08.634426 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:08.634548 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:08.635604 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:08.636645 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:08.637772 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:08.638277 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:08.638364 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:08.642072 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:08.642086 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:08.642102 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:08.642131 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:08.644881 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:08.645461 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:08.645467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:08.645470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:08.645589 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:08.645874 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:08.645877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:08.645879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:08.646189 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:08.646411 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:08.646756 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 15:04:08.646950 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":252,"threads_started":1}
I20260925 15:04:08.648639 23842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:08.728714 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 23719
I20260925 15:04:08.741796 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 23852
Sep 25 15:04:08 dist-test-slave-nsrc krb5kdc[23479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:04:08.756471 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:08.742752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50248 (local address 127.22.236.3:34307)
0925 15:04:08.743374 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:08.743381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:08.743409 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:04:08.743481 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:08.743489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:08.743707 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:08.743846 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:08.745021 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:08.745988 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:08.747166 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:08.747691 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:08.747791 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:08.751632 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:08.751643 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:08.751657 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:08.751688 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:08.754242 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:08.754825 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:08.754832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:08.754834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:08.754946 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:08.755210 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:08.755212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:08.755214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:08.755520 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:08.755737 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:08.755943 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 15:04:08.756127 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":317,"threads_started":1}
I20260925 15:04:08.757725 23975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:08.837754 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 23852
I20260925 15:04:08.850126 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 23492
I20260925 15:04:08.851248 23554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:08.952862 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 23492
2026-09-25T15:04:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3621 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:04:09 dist-test-slave-nsrc krb5kdc[24001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:04:09 dist-test-slave-nsrc krb5kdc[24001](info): set up 2 sockets
Sep 25 15:04:09 dist-test-slave-nsrc krb5kdc[24001](info): commencing operation
krb5kdc: starting...
W20260925 15:04:09.935822 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.011s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:04:10 dist-test-slave-nsrc krb5kdc[24001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:04:10.087427 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:39111
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:39111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:10.386428 24014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:10.386752 24014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:10.386859 24014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:10.394407 24014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:10.394526 24014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:10.394580 24014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:10.394634 24014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:10.394675 24014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:04:10.404633 24014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:39111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:39111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:10.405915 24014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:10.407132 24014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:10.418485 24020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:10.418516 24022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:10.418448 24019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:10.419407 24014 server_base.cc:1034] running on GCE node
I20260925 15:04:10.420202 24014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:10.421561 24014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:10.422758 24014 hybrid_clock.cc:648] HybridClock initialized: now 1790348650422714 us; error 37 us; skew 500 ppm
I20260925 15:04:10.423084 24014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:10 dist-test-slave-nsrc krb5kdc[24001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348650, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:10.426065 24014 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:04:10.427482 24014 webserver.cc:460] Webserver started at http://127.22.236.62:35395/ using document root <none> and password file <none>
I20260925 15:04:10.428181 24014 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:10.428303 24014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:10.428571 24014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:10.431427 24014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "fcf6ec4d72ad4f36933d55900413aa48"
format_stamp: "Formatted at 2026-09-25 15:04:10 on dist-test-slave-nsrc"
server_key: "aa6924ef52996fdb5d3cdb4a95881112"
server_key_iv: "b68480cb9fc416fee331cc5de4d3506b"
server_key_version: "encryptionkey@0"
I20260925 15:04:10.432116 24014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "fcf6ec4d72ad4f36933d55900413aa48"
format_stamp: "Formatted at 2026-09-25 15:04:10 on dist-test-slave-nsrc"
server_key: "aa6924ef52996fdb5d3cdb4a95881112"
server_key_iv: "b68480cb9fc416fee331cc5de4d3506b"
server_key_version: "encryptionkey@0"
I20260925 15:04:10.437099 24014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:04:10.440356 24029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:10.441627 24014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:04:10.441762 24014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "fcf6ec4d72ad4f36933d55900413aa48"
format_stamp: "Formatted at 2026-09-25 15:04:10 on dist-test-slave-nsrc"
server_key: "aa6924ef52996fdb5d3cdb4a95881112"
server_key_iv: "b68480cb9fc416fee331cc5de4d3506b"
server_key_version: "encryptionkey@0"
I20260925 15:04:10.441958 24014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:10.471807 24014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:10.475564 24014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:10.475800 24014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:10.491291 24014 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:39111
I20260925 15:04:10.491319 24080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:39111 every 8 connection(s)
I20260925 15:04:10.492587 24014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:10.496101 24081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:10.501634 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 24014
I20260925 15:04:10.501868 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:10.502609 23472 external_mini_cluster.cc:1428] Setting key 80430ec578b345f17716f160bfa23b38
I20260925 15:04:10.506842 24081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48: Bootstrap starting.
I20260925 15:04:10.510049 24081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:10.511116 24081 log.cc:824] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:10.514021 24081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48: No bootstrap required, opened a new log
Sep 25 15:04:10 dist-test-slave-nsrc krb5kdc[24001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:10.522995 24081 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf6ec4d72ad4f36933d55900413aa48" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39111 } }
I20260925 15:04:10.523483 24081 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:10.523587 24081 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf6ec4d72ad4f36933d55900413aa48, State: Initialized, Role: FOLLOWER
I20260925 15:04:10.524240 24081 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf6ec4d72ad4f36933d55900413aa48" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39111 } }
I20260925 15:04:10.524479 24081 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:10.524621 24081 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:10.524789 24081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:10.525094 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:10.504749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.22.236.62:39111)
0925 15:04:10.505496 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:10.505507 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:10.505544 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:04:10.507191 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:10.507213 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:10.507605 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:10.507954 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:10.509426 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:10.510716 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:10.512277 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:10.512917 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:10.513045 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:10.517720 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:10.517727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:10.517743 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:10.517776 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:10.520909 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:10.521671 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:10.521678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:10.521684 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:10.521857 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:10.522204 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:10.522207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:10.522209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:10.522711 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:10.522926 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:10.523474 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 15:04:10.523921 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":332,"threads_started":1}
I20260925 15:04:10.527026 24081 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf6ec4d72ad4f36933d55900413aa48" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39111 } }
I20260925 15:04:10.527488 24081 leader_election.cc:302] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcf6ec4d72ad4f36933d55900413aa48; no voters:
I20260925 15:04:10.528070 24081 leader_election.cc:288] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:10.528187 24086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:10.528756 24086 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [term 1 LEADER]: Becoming Leader. State: Replica: fcf6ec4d72ad4f36933d55900413aa48, State: Running, Role: LEADER
I20260925 15:04:10.529320 24086 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf6ec4d72ad4f36933d55900413aa48" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39111 } }
I20260925 15:04:10.530153 24081 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:10.532020 24087 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcf6ec4d72ad4f36933d55900413aa48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf6ec4d72ad4f36933d55900413aa48" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39111 } } }
I20260925 15:04:10.532006 24088 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcf6ec4d72ad4f36933d55900413aa48. Latest consensus state: current_term: 1 leader_uuid: "fcf6ec4d72ad4f36933d55900413aa48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf6ec4d72ad4f36933d55900413aa48" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39111 } } }
I20260925 15:04:10.532502 24087 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:10.532528 24088 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:10.533281 24093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:10.539896 24093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:10.548086 24093 catalog_manager.cc:1347] Generated new cluster ID: 605f1a31584e46638f80a1d91743c3cb
I20260925 15:04:10.548218 24093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:10.559888 24093 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:10.561689 24093 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:10.574761 24093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcf6ec4d72ad4f36933d55900413aa48: Generated new TSK 0
I20260925 15:04:10.575783 24093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:04:10.627038 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:39111
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:10.930285 24107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:10.930573 24107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:10.930711 24107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:10.937963 24107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:10.938083 24107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:10.938241 24107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:04:10.947892 24107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:39111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:10.949123 24107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:10.950402 24107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:10.964442 24112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:10.964463 24115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:10.964407 24113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:10.965199 24107 server_base.cc:1034] running on GCE node
I20260925 15:04:10.965766 24107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:10.966672 24107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:10.967864 24107 hybrid_clock.cc:648] HybridClock initialized: now 1790348650967811 us; error 50 us; skew 500 ppm
I20260925 15:04:10.968178 24107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:10 dist-test-slave-nsrc krb5kdc[24001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348650, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:10.971065 24107 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:04:10.972489 24107 webserver.cc:460] Webserver started at http://127.22.236.1:35219/ using document root <none> and password file <none>
I20260925 15:04:10.973134 24107 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:10.973242 24107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:10.973515 24107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:10.976239 24107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "e66f29e250d7411eb65029a330a539e9"
format_stamp: "Formatted at 2026-09-25 15:04:10 on dist-test-slave-nsrc"
server_key: "8b444dcaabbc4c2537c46efe95a581fb"
server_key_iv: "d7f20d42095f57ce775a7b9f769027c0"
server_key_version: "encryptionkey@0"
I20260925 15:04:10.976965 24107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "e66f29e250d7411eb65029a330a539e9"
format_stamp: "Formatted at 2026-09-25 15:04:10 on dist-test-slave-nsrc"
server_key: "8b444dcaabbc4c2537c46efe95a581fb"
server_key_iv: "d7f20d42095f57ce775a7b9f769027c0"
server_key_version: "encryptionkey@0"
I20260925 15:04:10.981644 24107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 15:04:10.984810 24122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:10.985997 24107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:10.986133 24107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "e66f29e250d7411eb65029a330a539e9"
format_stamp: "Formatted at 2026-09-25 15:04:10 on dist-test-slave-nsrc"
server_key: "8b444dcaabbc4c2537c46efe95a581fb"
server_key_iv: "d7f20d42095f57ce775a7b9f769027c0"
server_key_version: "encryptionkey@0"
I20260925 15:04:10.986289 24107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:10.998409 24107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:11.002300 24107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:11.002552 24107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:11.003422 24107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:11.004696 24107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:11.004769 24107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:11.004879 24107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:11.004930 24107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:11.034291 24107 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:41779
I20260925 15:04:11.034336 24234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:41779 every 8 connection(s)
I20260925 15:04:11.035557 24107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:04:11.042855 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 24107
I20260925 15:04:11.043047 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:04:11.043601 23472 external_mini_cluster.cc:1428] Setting key a16e67e08196660f1dee44d4bf8fabd1
Sep 25 15:04:11 dist-test-slave-nsrc krb5kdc[24001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348650, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:11.055068 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:11.037997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:44955 (local address 127.22.236.62:39111)
0925 15:04:11.038433 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:11.038437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:11.039349 (+ 912us) server_negotiation.cc:407] Connection header received
0925 15:04:11.041165 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:11.041171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:11.041259 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:11.041350 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:11.042836 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:11.043614 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:11.045333 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:11.045909 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:11.046025 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:11.049454 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:11.049461 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:11.049465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:11.049503 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:11.051996 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:11.052814 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:11.052818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:11.052820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:11.052959 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:11.053426 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:11.053430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:11.053432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:11.053694 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:11.053797 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:11.054651 (+ 854us) server_negotiation.cc:299] Negotiation successful
0925 15:04:11.054819 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":247,"threads_started":1}
I20260925 15:04:11.056537 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:11.038530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:39111 (local address 127.22.236.1:44955)
0925 15:04:11.039189 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:11.039239 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:11.040912 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:11.041587 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:11.041596 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:11.042109 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:11.042666 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:11.042692 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:11.043759 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:11.043763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:11.045196 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:11.045219 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:11.046108 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:11.046113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:11.046313 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:11.046971 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:11.046993 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:11.049234 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:11.052131 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:11.052161 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:11.052180 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:11.052704 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:11.053058 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:11.053063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:11.053066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:11.053299 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:11.053810 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:11.053818 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:11.054230 (+ 412us) client_negotiation.cc:769] Sending connection context
0925 15:04:11.054541 (+ 311us) client_negotiation.cc:240] Negotiation successful
0925 15:04:11.055080 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":258,"threads_started":1}
I20260925 15:04:11.058131 24235 heartbeater.cc:347] Connected to a master server at 127.22.236.62:39111
I20260925 15:04:11.058422 24235 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
I20260925 15:04:11.059110 24235 heartbeater.cc:511] Master 127.22.236.62:39111 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
I20260925 15:04:11.061236 24046 ts_manager.cc:194] Registered new tserver with Master: e66f29e250d7411eb65029a330a539e9 (127.22.236.1:41779)
I20260925 15:04:11.062628 24046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:44955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:04:11.078893 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:39111
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:11.379987 24241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:11.380335 24241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:11.380486 24241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:11.387458 24241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:11.387570 24241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:11.387720 24241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:04:11.397544 24241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:39111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:11.398799 24241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:11.400130 24241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:11.412323 24246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:11.412271 24249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:11.412267 24247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:11.413084 24241 server_base.cc:1034] running on GCE node
I20260925 15:04:11.413666 24241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:11.414556 24241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:11.415747 24241 hybrid_clock.cc:648] HybridClock initialized: now 1790348651415695 us; error 48 us; skew 500 ppm
I20260925 15:04:11.416087 24241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:11 dist-test-slave-nsrc krb5kdc[24001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348651, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:11.419494 24241 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:04:11.420941 24241 webserver.cc:460] Webserver started at http://127.22.236.2:36137/ using document root <none> and password file <none>
I20260925 15:04:11.421578 24241 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:11.421687 24241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:11.421979 24241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:11.424786 24241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "01d1754e5a774912a4f7d96d8636b46b"
format_stamp: "Formatted at 2026-09-25 15:04:11 on dist-test-slave-nsrc"
server_key: "bf1b203f2c61a5ec98ae8679e1d0823d"
server_key_iv: "63ad65cb07936348bd0232991a37e627"
server_key_version: "encryptionkey@0"
I20260925 15:04:11.425520 24241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "01d1754e5a774912a4f7d96d8636b46b"
format_stamp: "Formatted at 2026-09-25 15:04:11 on dist-test-slave-nsrc"
server_key: "bf1b203f2c61a5ec98ae8679e1d0823d"
server_key_iv: "63ad65cb07936348bd0232991a37e627"
server_key_version: "encryptionkey@0"
I20260925 15:04:11.430352 24241 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 15:04:11.433707 24256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:11.435075 24241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:04:11.435223 24241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "01d1754e5a774912a4f7d96d8636b46b"
format_stamp: "Formatted at 2026-09-25 15:04:11 on dist-test-slave-nsrc"
server_key: "bf1b203f2c61a5ec98ae8679e1d0823d"
server_key_iv: "63ad65cb07936348bd0232991a37e627"
server_key_version: "encryptionkey@0"
I20260925 15:04:11.435401 24241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:11.471489 24241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:11.475636 24241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:11.475934 24241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:11.476905 24241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:11.478219 24241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:11.478296 24241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:11.478423 24241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:11.478474 24241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:11.508159 24241 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:36229
I20260925 15:04:11.508205 24368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:36229 every 8 connection(s)
I20260925 15:04:11.509392 24241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:04:11.511871 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 24241
I20260925 15:04:11.512048 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:04:11.512486 23472 external_mini_cluster.cc:1428] Setting key 95310a15064b8fc6b284ac53cbfaa817
Sep 25 15:04:11 dist-test-slave-nsrc krb5kdc[24001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348651, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
I20260925 15:04:11.529608 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:11.511744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:50711 (local address 127.22.236.62:39111)
0925 15:04:11.511943 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:11.511949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:11.513173 (+ 1224us) server_negotiation.cc:407] Connection header received
0925 15:04:11.515324 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:11.515328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:11.515391 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:11.515471 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:11.517428 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:11.518106 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:11.519807 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:11.520425 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:11.520544 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:11.523782 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:11.523788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:11.523803 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:11.523840 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:11.526917 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:11.527575 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:11.527581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:11.527585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:11.527734 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:11.528127 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:11.528131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:11.528133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:11.528415 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:11.528562 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:11.529186 (+ 624us) server_negotiation.cc:299] Negotiation successful
0925 15:04:11.529362 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
I20260925 15:04:11.530781 24371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:11.512256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:39111 (local address 127.22.236.2:50711)
0925 15:04:11.513014 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:11.513061 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:11.515029 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:11.515739 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:11.515751 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:11.516506 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:11.517283 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:11.517306 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:11.518238 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:11.518244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:11.519659 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:11.519671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:11.520655 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:11.520663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:11.520888 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:11.521497 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:11.521523 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:11.523546 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:11.527046 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:11.527053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:11.527066 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:11.527455 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:11.527861 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:11.527864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:11.527867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:11.528041 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:11.528525 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:11.528532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:11.528876 (+ 344us) client_negotiation.cc:769] Sending connection context
0925 15:04:11.529104 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 15:04:11.529530 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":281,"threads_started":1}
I20260925 15:04:11.532291 24369 heartbeater.cc:347] Connected to a master server at 127.22.236.62:39111
I20260925 15:04:11.532618 24369 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:11.533439 24369 heartbeater.cc:511] Master 127.22.236.62:39111 requested a full tablet report, sending...
I20260925 15:04:11.535029 24046 ts_manager.cc:194] Registered new tserver with Master: 01d1754e5a774912a4f7d96d8636b46b (127.22.236.2:36229)
I20260925 15:04:11.535995 24046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:50711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:04:11.550428 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:39111
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:11.853443 24374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:11.853721 24374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:11.853859 24374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:11.860795 24374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:11.860906 24374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:11.861061 24374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:04:11.870383 24374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:39111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:11.871587 24374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:11.872876 24374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:11.884654 24380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:11.884630 24382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:11.884698 24379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:11.885458 24374 server_base.cc:1034] running on GCE node
I20260925 15:04:11.886008 24374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:11.886863 24374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:11.888063 24374 hybrid_clock.cc:648] HybridClock initialized: now 1790348651888015 us; error 41 us; skew 500 ppm
I20260925 15:04:11.888375 24374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:11 dist-test-slave-nsrc krb5kdc[24001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348651, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:11.897092 24374 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:04:11.898520 24374 webserver.cc:460] Webserver started at http://127.22.236.3:43807/ using document root <none> and password file <none>
I20260925 15:04:11.899158 24374 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:11.899278 24374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:11.899554 24374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:11.902356 24374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a"
format_stamp: "Formatted at 2026-09-25 15:04:11 on dist-test-slave-nsrc"
server_key: "1239c37bef475e269a3cb51e876aebf5"
server_key_iv: "7f81124a7cdf07d8426fe9b60fde4127"
server_key_version: "encryptionkey@0"
I20260925 15:04:11.903088 24374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a"
format_stamp: "Formatted at 2026-09-25 15:04:11 on dist-test-slave-nsrc"
server_key: "1239c37bef475e269a3cb51e876aebf5"
server_key_iv: "7f81124a7cdf07d8426fe9b60fde4127"
server_key_version: "encryptionkey@0"
I20260925 15:04:11.908046 24374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260925 15:04:11.911301 24389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:11.912595 24374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:11.912737 24374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a"
format_stamp: "Formatted at 2026-09-25 15:04:11 on dist-test-slave-nsrc"
server_key: "1239c37bef475e269a3cb51e876aebf5"
server_key_iv: "7f81124a7cdf07d8426fe9b60fde4127"
server_key_version: "encryptionkey@0"
I20260925 15:04:11.912917 24374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:11.929028 24374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:11.934803 24374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:11.935103 24374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:11.936369 24374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:11.937981 24374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:11.938071 24374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:11.938243 24374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:11.938315 24374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 15:04:11.971789 24374 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:38781
I20260925 15:04:11.971803 24501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:38781 every 8 connection(s)
I20260925 15:04:11.973358 24374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:04:11.973630 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 24374
I20260925 15:04:11.973816 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:04:11.974310 23472 external_mini_cluster.cc:1428] Setting key 3813e951c56d740cb0169f34ad40c1df
Sep 25 15:04:11 dist-test-slave-nsrc krb5kdc[24001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348651, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:11.993081 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:11.977710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:37509 (local address 127.22.236.62:39111)
0925 15:04:11.977902 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:11.977908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:11.978193 (+ 285us) server_negotiation.cc:407] Connection header received
0925 15:04:11.980395 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:11.980399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:11.980459 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:11.980540 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:11.981905 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:11.982588 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:11.984014 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:11.984514 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:11.984600 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:11.987618 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:11.987622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:11.987625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:11.987655 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:11.990388 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:11.991095 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:11.991100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:11.991104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:11.991243 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:11.991627 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:11.991630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:11.991632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:11.991911 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:11.992013 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:11.992673 (+ 660us) server_negotiation.cc:299] Negotiation successful
0925 15:04:11.992855 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:04:11.994352 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:11.977344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:39111 (local address 127.22.236.3:37509)
0925 15:04:11.978048 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:11.978093 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:11.980068 (+ 1975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:11.980740 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:11.980749 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:11.981244 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:11.981775 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:11.981791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:11.982699 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:11.982702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:11.983915 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:11.983925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:11.984700 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:11.984705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:11.984902 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:11.985432 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:11.985451 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:11.987428 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:11.990518 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:11.990528 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:11.990545 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:11.991000 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:11.991356 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:11.991361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:11.991365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:11.991540 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:11.992017 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:11.992023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:11.992365 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 15:04:11.992578 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 15:04:11.992932 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":455,"threads_started":1}
I20260925 15:04:11.995587 24502 heartbeater.cc:347] Connected to a master server at 127.22.236.62:39111
I20260925 15:04:11.995880 24502 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:11.996573 24502 heartbeater.cc:511] Master 127.22.236.62:39111 requested a full tablet report, sending...
I20260925 15:04:11.997898 24046 ts_manager.cc:194] Registered new tserver with Master: 6dd5a555556d4c4a9aaad2752f7c2d3a (127.22.236.3:38781)
I20260925 15:04:11.998629 24046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:37509
I20260925 15:04:12.003280 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:04:12.020845 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.009185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.22.236.62:39111)
0925 15:04:12.009342 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:12.009346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:12.009832 (+ 486us) server_negotiation.cc:407] Connection header received
0925 15:04:12.009961 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:12.009965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:12.010024 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:12.010103 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:12.011404 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.012088 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.013332 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.013723 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.013824 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.016458 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:12.016462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:12.016465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:12.016494 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:12.018762 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:12.019323 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:12.019327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:12.019329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:12.019443 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:12.019700 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:12.019703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:12.019705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:12.019994 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:12.020183 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:12.020447 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 15:04:12.020588 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:04:12.026643 24046 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:04:12.033449 24046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:04:12.065537 24235 heartbeater.cc:503] Master 127.22.236.62:39111 was elected leader, sending a full tablet report...
I20260925 15:04:12.075994 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.045669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:36229 (local address 127.0.0.1:59588)
0925 15:04:12.068248 (+ 22579us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:12.068268 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:12.068379 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:12.069285 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:12.069289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:12.069318 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:12.070275 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:12.070289 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.071744 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.071749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:12.073791 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:12.073802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.075031 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.075038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:12.075203 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.075244 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 15:04:12.075384 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 15:04:12.075566 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22447,"mutex_wait_us":2,"thread_start_us":20100,"threads_started":1}
I20260925 15:04:12.076609 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.045286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.22.236.3:38781)
0925 15:04:12.068008 (+ 22722us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:12.068014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:12.068040 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:04:12.068115 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:12.068121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:12.068401 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:12.068582 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:12.069653 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.071369 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.073867 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.075221 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.075342 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.075536 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:12.076083 (+ 547us) server_negotiation.cc:299] Negotiation successful
0925 15:04:12.076268 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22585,"thread_start_us":22447,"threads_started":1}
I20260925 15:04:12.076910 24519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.046180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59588 (local address 127.22.236.2:36229)
0925 15:04:12.067105 (+ 20925us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:12.067116 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:12.068462 (+ 1346us) server_negotiation.cc:407] Connection header received
0925 15:04:12.068656 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:12.068663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:12.068895 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:12.069062 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:12.070449 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.071571 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.073941 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.074823 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.074949 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.075148 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:12.076469 (+ 1321us) server_negotiation.cc:299] Negotiation successful
0925 15:04:12.076631 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20733,"thread_start_us":20612,"threads_started":1}
I20260925 15:04:12.077356 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.045169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:38781 (local address 127.0.0.1:36474)
0925 15:04:12.066246 (+ 21077us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:12.066295 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:12.066513 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:12.068874 (+ 2361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:12.068882 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:12.068916 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:12.069425 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:12.069438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.071515 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.071520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:12.073722 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:12.073732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.075420 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.075426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:12.075592 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.075628 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:04:12.075719 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:04:12.075831 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20769,"mutex_wait_us":45,"thread_start_us":21981,"threads_started":1}
I20260925 15:04:12.078992 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.045204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:41779 (local address 127.0.0.1:43856)
0925 15:04:12.067447 (+ 22243us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:12.067467 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:12.068284 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:12.069117 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:12.069121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:12.069140 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:12.070335 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:12.070349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.072062 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.072067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:12.075747 (+ 3680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:12.075760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.077759 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.077765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:12.077924 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.077953 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:04:12.078031 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:04:12.078150 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22105,"mutex_wait_us":94,"thread_start_us":22801,"threads_started":1}
I20260925 15:04:12.079195 24518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.045749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43856 (local address 127.22.236.1:41779)
0925 15:04:12.067787 (+ 22038us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:12.067796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:12.067830 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:04:12.068485 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:12.068494 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:12.068709 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:12.068888 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:12.070504 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.071927 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.075921 (+ 3994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.077235 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.077354 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.077556 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:12.078758 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0925 15:04:12.078913 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21882,"thread_start_us":21738,"threads_started":1}
I20260925 15:04:12.081624 24437 tablet_service.cc:1467] Processing CreateTablet for tablet 6fae689ba6434b3dae4b8222acf5d879 (DEFAULT_TABLE table=test-table [id=a6694a2bbd4343b9881a76190e2c5f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:12.082278 24170 tablet_service.cc:1467] Processing CreateTablet for tablet 6fae689ba6434b3dae4b8222acf5d879 (DEFAULT_TABLE table=test-table [id=a6694a2bbd4343b9881a76190e2c5f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:12.082821 24437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fae689ba6434b3dae4b8222acf5d879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:12.081574 24304 tablet_service.cc:1467] Processing CreateTablet for tablet 6fae689ba6434b3dae4b8222acf5d879 (DEFAULT_TABLE table=test-table [id=a6694a2bbd4343b9881a76190e2c5f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:12.083559 24170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fae689ba6434b3dae4b8222acf5d879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:12.083586 24304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fae689ba6434b3dae4b8222acf5d879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:12.093549 24520 tablet_bootstrap.cc:492] T 6fae689ba6434b3dae4b8222acf5d879 P 6dd5a555556d4c4a9aaad2752f7c2d3a: Bootstrap starting.
I20260925 15:04:12.096508 24520 tablet_bootstrap.cc:654] T 6fae689ba6434b3dae4b8222acf5d879 P 6dd5a555556d4c4a9aaad2752f7c2d3a: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:12.097796 24522 tablet_bootstrap.cc:492] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9: Bootstrap starting.
I20260925 15:04:12.098910 24520 log.cc:824] T 6fae689ba6434b3dae4b8222acf5d879 P 6dd5a555556d4c4a9aaad2752f7c2d3a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:12.099042 24521 tablet_bootstrap.cc:492] T 6fae689ba6434b3dae4b8222acf5d879 P 01d1754e5a774912a4f7d96d8636b46b: Bootstrap starting.
I20260925 15:04:12.101138 24522 tablet_bootstrap.cc:654] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:12.101751 24520 tablet_bootstrap.cc:492] T 6fae689ba6434b3dae4b8222acf5d879 P 6dd5a555556d4c4a9aaad2752f7c2d3a: No bootstrap required, opened a new log
I20260925 15:04:12.102005 24520 ts_tablet_manager.cc:1397] T 6fae689ba6434b3dae4b8222acf5d879 P 6dd5a555556d4c4a9aaad2752f7c2d3a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 15:04:12.102298 24522 log.cc:824] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:12.102381 24521 tablet_bootstrap.cc:654] T 6fae689ba6434b3dae4b8222acf5d879 P 01d1754e5a774912a4f7d96d8636b46b: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:12.103466 24521 log.cc:824] T 6fae689ba6434b3dae4b8222acf5d879 P 01d1754e5a774912a4f7d96d8636b46b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:12.105742 24522 tablet_bootstrap.cc:492] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9: No bootstrap required, opened a new log
I20260925 15:04:12.106002 24522 ts_tablet_manager.cc:1397] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260925 15:04:12.109541 24520 raft_consensus.cc:348] T 6fae689ba6434b3dae4b8222acf5d879 P 6dd5a555556d4c4a9aaad2752f7c2d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38781 } } peers { permanent_uuid: "e66f29e250d7411eb65029a330a539e9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41779 } } peers { permanent_uuid: "01d1754e5a774912a4f7d96d8636b46b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36229 } }
I20260925 15:04:12.109872 24520 raft_consensus.cc:374] T 6fae689ba6434b3dae4b8222acf5d879 P 6dd5a555556d4c4a9aaad2752f7c2d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:12.109956 24520 raft_consensus.cc:729] T 6fae689ba6434b3dae4b8222acf5d879 P 6dd5a555556d4c4a9aaad2752f7c2d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd5a555556d4c4a9aaad2752f7c2d3a, State: Initialized, Role: FOLLOWER
I20260925 15:04:12.110494 24520 consensus_queue.cc:260] T 6fae689ba6434b3dae4b8222acf5d879 P 6dd5a555556d4c4a9aaad2752f7c2d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38781 } } peers { permanent_uuid: "e66f29e250d7411eb65029a330a539e9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41779 } } peers { permanent_uuid: "01d1754e5a774912a4f7d96d8636b46b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36229 } }
I20260925 15:04:12.112809 24502 heartbeater.cc:503] Master 127.22.236.62:39111 was elected leader, sending a full tablet report...
I20260925 15:04:12.113498 24520 ts_tablet_manager.cc:1428] T 6fae689ba6434b3dae4b8222acf5d879 P 6dd5a555556d4c4a9aaad2752f7c2d3a: Time spent starting tablet: real 0.011s user 0.005s sys 0.007s
I20260925 15:04:12.116209 24521 tablet_bootstrap.cc:492] T 6fae689ba6434b3dae4b8222acf5d879 P 01d1754e5a774912a4f7d96d8636b46b: No bootstrap required, opened a new log
I20260925 15:04:12.116461 24521 ts_tablet_manager.cc:1397] T 6fae689ba6434b3dae4b8222acf5d879 P 01d1754e5a774912a4f7d96d8636b46b: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260925 15:04:12.116950 24522 raft_consensus.cc:348] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38781 } } peers { permanent_uuid: "e66f29e250d7411eb65029a330a539e9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41779 } } peers { permanent_uuid: "01d1754e5a774912a4f7d96d8636b46b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36229 } }
I20260925 15:04:12.117355 24522 raft_consensus.cc:374] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:12.117445 24522 raft_consensus.cc:729] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e66f29e250d7411eb65029a330a539e9, State: Initialized, Role: FOLLOWER
I20260925 15:04:12.118000 24522 consensus_queue.cc:260] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38781 } } peers { permanent_uuid: "e66f29e250d7411eb65029a330a539e9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41779 } } peers { permanent_uuid: "01d1754e5a774912a4f7d96d8636b46b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36229 } }
I20260925 15:04:12.121354 24522 ts_tablet_manager.cc:1428] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260925 15:04:12.125061 24521 raft_consensus.cc:348] T 6fae689ba6434b3dae4b8222acf5d879 P 01d1754e5a774912a4f7d96d8636b46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38781 } } peers { permanent_uuid: "e66f29e250d7411eb65029a330a539e9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41779 } } peers { permanent_uuid: "01d1754e5a774912a4f7d96d8636b46b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36229 } }
I20260925 15:04:12.125367 24521 raft_consensus.cc:374] T 6fae689ba6434b3dae4b8222acf5d879 P 01d1754e5a774912a4f7d96d8636b46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:12.125453 24521 raft_consensus.cc:729] T 6fae689ba6434b3dae4b8222acf5d879 P 01d1754e5a774912a4f7d96d8636b46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d1754e5a774912a4f7d96d8636b46b, State: Initialized, Role: FOLLOWER
I20260925 15:04:12.125928 24521 consensus_queue.cc:260] T 6fae689ba6434b3dae4b8222acf5d879 P 01d1754e5a774912a4f7d96d8636b46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38781 } } peers { permanent_uuid: "e66f29e250d7411eb65029a330a539e9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41779 } } peers { permanent_uuid: "01d1754e5a774912a4f7d96d8636b46b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36229 } }
I20260925 15:04:12.126780 24369 heartbeater.cc:503] Master 127.22.236.62:39111 was elected leader, sending a full tablet report...
I20260925 15:04:12.127192 24521 ts_tablet_manager.cc:1428] T 6fae689ba6434b3dae4b8222acf5d879 P 01d1754e5a774912a4f7d96d8636b46b: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260925 15:04:12.153275 24527 raft_consensus.cc:482] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:04:12.153566 24527 raft_consensus.cc:504] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38781 } } peers { permanent_uuid: "e66f29e250d7411eb65029a330a539e9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41779 } } peers { permanent_uuid: "01d1754e5a774912a4f7d96d8636b46b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36229 } }
I20260925 15:04:12.155131 24527 leader_election.cc:288] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dd5a555556d4c4a9aaad2752f7c2d3a (127.22.236.3:38781), 01d1754e5a774912a4f7d96d8636b46b (127.22.236.2:36229)
I20260925 15:04:12.163810 24519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.155480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:53487 (local address 127.22.236.2:36229)
0925 15:04:12.155663 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:12.155668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:12.156369 (+ 701us) server_negotiation.cc:407] Connection header received
0925 15:04:12.156533 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:12.156538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:12.156613 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:12.156712 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:12.157448 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.158468 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.161666 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.162845 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.163010 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.163080 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:12.163430 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 15:04:12.163553 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:04:12.163810 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.155635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:41647 (local address 127.22.236.3:38781)
0925 15:04:12.155829 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:12.155834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:12.156364 (+ 530us) server_negotiation.cc:407] Connection header received
0925 15:04:12.156561 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:12.156566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:12.156630 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:12.156712 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:12.157370 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.158474 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.161669 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.162845 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.163005 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.163080 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:12.163430 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 15:04:12.163553 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:04:12.164181 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.155382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:38781 (local address 127.22.236.1:41647)
0925 15:04:12.156230 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:12.156249 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:12.156447 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:12.156742 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:12.156746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:12.156789 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:12.157209 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:12.157224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.158594 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.158599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:12.161525 (+ 2926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:12.161535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.163089 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.163095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:12.163275 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.163317 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:04:12.163399 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:04:12.163591 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":130,"spinlock_wait_cycles":39424,"thread_start_us":618,"threads_started":1}
I20260925 15:04:12.164214 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.155505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:36229 (local address 127.22.236.1:53487)
0925 15:04:12.156270 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:12.156285 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:12.156408 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:12.156742 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:12.156746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:12.156792 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:12.157306 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:12.157317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.158594 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.158599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:12.161507 (+ 2908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:12.161527 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.163076 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.163083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:12.163279 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.163309 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:04:12.163400 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:04:12.163595 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":245,"spinlock_wait_cycles":21888,"thread_start_us":359,"threads_started":1}
I20260925 15:04:12.165234 24324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fae689ba6434b3dae4b8222acf5d879" candidate_uuid: "e66f29e250d7411eb65029a330a539e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d1754e5a774912a4f7d96d8636b46b" is_pre_election: true
I20260925 15:04:12.165242 24457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fae689ba6434b3dae4b8222acf5d879" candidate_uuid: "e66f29e250d7411eb65029a330a539e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a" is_pre_election: true
I20260925 15:04:12.165706 24457 raft_consensus.cc:2454] T 6fae689ba6434b3dae4b8222acf5d879 P 6dd5a555556d4c4a9aaad2752f7c2d3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e66f29e250d7411eb65029a330a539e9 in term 0.
I20260925 15:04:12.165709 24324 raft_consensus.cc:2454] T 6fae689ba6434b3dae4b8222acf5d879 P 01d1754e5a774912a4f7d96d8636b46b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e66f29e250d7411eb65029a330a539e9 in term 0.
I20260925 15:04:12.166373 24125 leader_election.cc:302] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01d1754e5a774912a4f7d96d8636b46b, 6dd5a555556d4c4a9aaad2752f7c2d3a, e66f29e250d7411eb65029a330a539e9; no voters:
I20260925 15:04:12.166738 24527 raft_consensus.cc:2781] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:04:12.166862 24527 raft_consensus.cc:482] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:04:12.166952 24527 raft_consensus.cc:3037] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:12.169059 24527 raft_consensus.cc:504] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38781 } } peers { permanent_uuid: "e66f29e250d7411eb65029a330a539e9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41779 } } peers { permanent_uuid: "01d1754e5a774912a4f7d96d8636b46b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36229 } }
I20260925 15:04:12.169584 24527 leader_election.cc:288] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [CANDIDATE]: Term 1 election: Requested vote from peers 6dd5a555556d4c4a9aaad2752f7c2d3a (127.22.236.3:38781), 01d1754e5a774912a4f7d96d8636b46b (127.22.236.2:36229)
I20260925 15:04:12.169963 24457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fae689ba6434b3dae4b8222acf5d879" candidate_uuid: "e66f29e250d7411eb65029a330a539e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a"
I20260925 15:04:12.170039 24324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fae689ba6434b3dae4b8222acf5d879" candidate_uuid: "e66f29e250d7411eb65029a330a539e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d1754e5a774912a4f7d96d8636b46b"
I20260925 15:04:12.170150 24457 raft_consensus.cc:3037] T 6fae689ba6434b3dae4b8222acf5d879 P 6dd5a555556d4c4a9aaad2752f7c2d3a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:12.170207 24324 raft_consensus.cc:3037] T 6fae689ba6434b3dae4b8222acf5d879 P 01d1754e5a774912a4f7d96d8636b46b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:12.173235 24457 raft_consensus.cc:2454] T 6fae689ba6434b3dae4b8222acf5d879 P 6dd5a555556d4c4a9aaad2752f7c2d3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e66f29e250d7411eb65029a330a539e9 in term 1.
I20260925 15:04:12.173565 24324 raft_consensus.cc:2454] T 6fae689ba6434b3dae4b8222acf5d879 P 01d1754e5a774912a4f7d96d8636b46b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e66f29e250d7411eb65029a330a539e9 in term 1.
I20260925 15:04:12.173593 24125 leader_election.cc:302] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dd5a555556d4c4a9aaad2752f7c2d3a, e66f29e250d7411eb65029a330a539e9; no voters:
I20260925 15:04:12.173839 24527 raft_consensus.cc:2781] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:12.174273 24527 raft_consensus.cc:686] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [term 1 LEADER]: Becoming Leader. State: Replica: e66f29e250d7411eb65029a330a539e9, State: Running, Role: LEADER
I20260925 15:04:12.174690 24527 consensus_queue.cc:237] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38781 } } peers { permanent_uuid: "e66f29e250d7411eb65029a330a539e9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41779 } } peers { permanent_uuid: "01d1754e5a774912a4f7d96d8636b46b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36229 } }
I20260925 15:04:12.179009 24046 catalog_manager.cc:5495] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e66f29e250d7411eb65029a330a539e9 (127.22.236.1). New cstate: current_term: 1 leader_uuid: "e66f29e250d7411eb65029a330a539e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd5a555556d4c4a9aaad2752f7c2d3a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e66f29e250d7411eb65029a330a539e9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01d1754e5a774912a4f7d96d8636b46b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36229 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:04:12 dist-test-slave-nsrc krb5kdc[24001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:04:12.205940 24518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.191769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.22.236.1:41779)
0925 15:04:12.191931 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:12.191935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:12.192445 (+ 510us) server_negotiation.cc:407] Connection header received
0925 15:04:12.192604 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:12.192609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:12.192674 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:12.192772 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:12.194164 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.194854 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.196066 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.196528 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.196635 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.200752 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:12.200766 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:12.200782 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:12.200815 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:12.203282 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:12.203900 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:12.203909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:12.203911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:12.204062 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:12.204447 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:12.204451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:12.204455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:12.204815 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:12.205089 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:12.205393 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 15:04:12.205635 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 15:04:12 dist-test-slave-nsrc krb5kdc[24001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348652, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260925 15:04:12.227694 24503 tablet.cc:2369] T 6fae689ba6434b3dae4b8222acf5d879 P 6dd5a555556d4c4a9aaad2752f7c2d3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:04:12 dist-test-slave-nsrc krb5kdc[24001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348652, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:04:12.246240 24518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.233039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.22.236.1:41779)
0925 15:04:12.233190 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:12.233195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:12.233549 (+ 354us) server_negotiation.cc:407] Connection header received
0925 15:04:12.233688 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:12.233692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:12.233750 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:12.233837 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:12.235092 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.235893 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.237126 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.237507 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.237593 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.241396 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:12.241401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:12.241404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:12.241435 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:12.244164 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:12.244657 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:12.244660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:12.244662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:12.244825 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:12.245168 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:12.245171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:12.245173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:12.245431 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:12.245621 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:12.245796 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 15:04:12.245985 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 15:04:12.261170 24370 tablet.cc:2369] T 6fae689ba6434b3dae4b8222acf5d879 P 01d1754e5a774912a4f7d96d8636b46b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:04:12 dist-test-slave-nsrc krb5kdc[24001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:04:12 dist-test-slave-nsrc krb5kdc[24001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:04:12.284247 24518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.271639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.22.236.1:41779)
0925 15:04:12.271840 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:12.271846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:12.272084 (+ 238us) server_negotiation.cc:407] Connection header received
0925 15:04:12.272288 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:12.272292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:12.272366 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:12.272448 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:12.273734 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.274466 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.275642 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.276059 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.276184 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.279735 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:12.279740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:12.279743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:12.279775 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:12.282031 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:12.282602 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:12.282606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:12.282608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:12.282719 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:12.283070 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:12.283073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:12.283076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:12.283343 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:12.283523 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:12.283735 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 15:04:12.283902 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 15:04:12.286275 24150 tablet_service.cc:2908] Rejecting scan request for tablet 6fae689ba6434b3dae4b8222acf5d879: Uninitialized: clean time has not yet been initialized
W20260925 15:04:12.287709 24236 tablet.cc:2369] T 6fae689ba6434b3dae4b8222acf5d879 P e66f29e250d7411eb65029a330a539e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:04:12.289363 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 24107
I20260925 15:04:12.300199 24518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.290146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.22.236.1:41779)
0925 15:04:12.290347 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:12.290354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:12.290617 (+ 263us) server_negotiation.cc:407] Connection header received
0925 15:04:12.290750 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:12.290753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:12.290825 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:12.290896 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:12.291882 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.292598 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.293680 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.294047 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.294128 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.295985 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:12.295989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:12.295992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:12.296019 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:12.298168 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:12.298723 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:12.298726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:12.298728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:12.298823 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:12.299097 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:12.299099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:12.299101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:12.299337 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:12.299500 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:12.299701 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 15:04:12.299869 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:04:12.300753 24230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:12.388363 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 24107
I20260925 15:04:12.401648 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 24241
Sep 25 15:04:12 dist-test-slave-nsrc krb5kdc[24001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:04:12.414358 24519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.402411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.22.236.2:36229)
0925 15:04:12.402556 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:12.402560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:12.402578 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:04:12.402632 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:12.402636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:12.402701 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:12.402779 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:12.403827 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.404585 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.405738 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.406108 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.406193 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.409762 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:12.409774 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:12.409790 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:12.409834 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:12.412318 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:12.412870 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:12.412876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:12.412878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:12.412990 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:12.413244 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:12.413247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:12.413249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:12.413569 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:12.413807 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:12.413976 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 15:04:12.414123 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:04:12.414995 24364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:12.501071 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 24241
I20260925 15:04:12.514192 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 24374
Sep 25 15:04:12 dist-test-slave-nsrc krb5kdc[24001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:04:12.528023 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:12.514913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.22.236.3:38781)
0925 15:04:12.515104 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:12.515109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:12.515128 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:04:12.515271 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:12.515276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:12.515350 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:12.515432 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:12.516752 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.517511 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.518802 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:12.519196 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:12.519301 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:12.523233 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:12.523247 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:12.523262 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:12.523301 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:12.525914 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:12.526511 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:12.526517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:12.526520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:12.526637 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:12.526900 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:12.526903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:12.526905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:12.527230 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:12.527437 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:12.527629 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 15:04:12.527779 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:04:12.528645 24497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:12.613716 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 24374
I20260925 15:04:12.626362 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 24014
I20260925 15:04:12.627439 24076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:12.731026 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 24014
2026-09-25T15:04:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3778 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:04:12 dist-test-slave-nsrc krb5kdc[24550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:04:12 dist-test-slave-nsrc krb5kdc[24550](info): set up 2 sockets
Sep 25 15:04:12 dist-test-slave-nsrc krb5kdc[24550](info): commencing operation
krb5kdc: starting...
W20260925 15:04:13.712034 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:04:13 dist-test-slave-nsrc krb5kdc[24550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:04:13.870666 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:33711
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:40827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:33711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:14.176983 24563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:14.177259 24563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:14.177387 24563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:14.184646 24563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:14.184769 24563 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:04:14.184815 24563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:14.184896 24563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:14.184950 24563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:14.185007 24563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:04:14.194898 24563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:40827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:33711
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:33711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:14.196127 24563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:14.197477 24563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:14.208484 24568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:14.208539 24571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:14.208514 24569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:14.209565 24563 server_base.cc:1034] running on GCE node
I20260925 15:04:14.210336 24563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:14.211669 24563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:14.212895 24563 hybrid_clock.cc:648] HybridClock initialized: now 1790348654212825 us; error 46 us; skew 500 ppm
I20260925 15:04:14.213217 24563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:14 dist-test-slave-nsrc krb5kdc[24550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348654, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:14.216298 24563 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:04:14.217720 24563 webserver.cc:460] Webserver started at http://127.22.236.62:35315/ using document root <none> and password file <none>
I20260925 15:04:14.218380 24563 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:14.218473 24563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:14.218772 24563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:14.221583 24563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "c8c0bd9e4c434f4bb6f0e649b07dd654"
format_stamp: "Formatted at 2026-09-25 15:04:14 on dist-test-slave-nsrc"
server_key: "52e0646b88e8a4731a982baa8fd3209a"
server_key_iv: "ebffdf290be55b9ad2e713c14d300332"
server_key_version: "encryptionkey@0"
I20260925 15:04:14.222292 24563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "c8c0bd9e4c434f4bb6f0e649b07dd654"
format_stamp: "Formatted at 2026-09-25 15:04:14 on dist-test-slave-nsrc"
server_key: "52e0646b88e8a4731a982baa8fd3209a"
server_key_iv: "ebffdf290be55b9ad2e713c14d300332"
server_key_version: "encryptionkey@0"
I20260925 15:04:14.227178 24563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 15:04:14.230696 24578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:14.231943 24563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:04:14.232087 24563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "c8c0bd9e4c434f4bb6f0e649b07dd654"
format_stamp: "Formatted at 2026-09-25 15:04:14 on dist-test-slave-nsrc"
server_key: "52e0646b88e8a4731a982baa8fd3209a"
server_key_iv: "ebffdf290be55b9ad2e713c14d300332"
server_key_version: "encryptionkey@0"
I20260925 15:04:14.232295 24563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:14.257726 24563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:14.261978 24563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:14.262225 24563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:14.281188 24563 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:33711
I20260925 15:04:14.281207 24639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:33711 every 8 connection(s)
I20260925 15:04:14.282437 24563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:14.285969 24640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:14.286000 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 24563
I20260925 15:04:14.286187 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:14.286787 23472 external_mini_cluster.cc:1428] Setting key 78ca4e41a2c28e5930b20180a5f90ab0
I20260925 15:04:14.298856 24640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654: Bootstrap starting.
Sep 25 15:04:14 dist-test-slave-nsrc krb5kdc[24550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:14.302554 24640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:14.303557 24640 log.cc:824] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:14.306505 24640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654: No bootstrap required, opened a new log
I20260925 15:04:14.309984 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:14.288661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.22.236.62:33711)
0925 15:04:14.289393 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:14.289408 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:14.289448 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:04:14.291434 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:14.291457 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:14.291871 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:14.292294 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:14.293973 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:14.295235 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:14.296991 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:14.297646 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:14.297769 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:14.302367 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:14.302375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:14.302389 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:14.302427 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:14.305712 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:14.306335 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:14.306342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:14.306350 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:14.306554 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:14.306924 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:14.306933 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:14.306936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:14.307601 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:14.307872 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:14.308252 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 15:04:14.308698 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":357,"threads_started":1}
I20260925 15:04:14.315088 24640 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0bd9e4c434f4bb6f0e649b07dd654" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33711 } }
I20260925 15:04:14.315389 24640 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:14.315459 24640 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c0bd9e4c434f4bb6f0e649b07dd654, State: Initialized, Role: FOLLOWER
I20260925 15:04:14.315920 24640 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0bd9e4c434f4bb6f0e649b07dd654" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33711 } }
I20260925 15:04:14.316107 24640 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:14.316253 24640 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:14.316378 24640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:14.318495 24640 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0bd9e4c434f4bb6f0e649b07dd654" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33711 } }
I20260925 15:04:14.318917 24640 leader_election.cc:302] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8c0bd9e4c434f4bb6f0e649b07dd654; no voters:
I20260925 15:04:14.319380 24640 leader_election.cc:288] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:14.319511 24645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:14.320119 24645 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [term 1 LEADER]: Becoming Leader. State: Replica: c8c0bd9e4c434f4bb6f0e649b07dd654, State: Running, Role: LEADER
I20260925 15:04:14.320578 24645 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0bd9e4c434f4bb6f0e649b07dd654" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33711 } }
I20260925 15:04:14.321251 24640 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:14.323176 24647 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8c0bd9e4c434f4bb6f0e649b07dd654. Latest consensus state: current_term: 1 leader_uuid: "c8c0bd9e4c434f4bb6f0e649b07dd654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0bd9e4c434f4bb6f0e649b07dd654" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33711 } } }
I20260925 15:04:14.323200 24646 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8c0bd9e4c434f4bb6f0e649b07dd654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c0bd9e4c434f4bb6f0e649b07dd654" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33711 } } }
I20260925 15:04:14.323581 24647 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:14.323580 24646 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:14.326186 24650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:14.330813 24650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:14.339291 24650 catalog_manager.cc:1347] Generated new cluster ID: 133c4b75720a4ebd9022fc166f5fc2b5
I20260925 15:04:14.339386 24650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:14.346629 24650 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:14.348032 24650 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:14.353924 24650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654: Generated new TSK 0
I20260925 15:04:14.354619 24650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:04:14.397540 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33711
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:40827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:14.718191 24666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:14.718465 24666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:14.718559 24666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:14.725687 24666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:14.725798 24666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:14.725880 24666 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:04:14.725986 24666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:04:14.735776 24666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:40827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.236.62:33711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:14.737130 24666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:14.738428 24666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:14.751535 24674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:14.751556 24672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:14.751645 24671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:14.752331 24666 server_base.cc:1034] running on GCE node
I20260925 15:04:14.752875 24666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:14.753724 24666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:14.754966 24666 hybrid_clock.cc:648] HybridClock initialized: now 1790348654754897 us; error 43 us; skew 500 ppm
I20260925 15:04:14.755280 24666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:14 dist-test-slave-nsrc krb5kdc[24550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348654, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:14.758356 24666 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:04:14.759806 24666 webserver.cc:460] Webserver started at http://127.22.236.1:39141/ using document root <none> and password file <none>
I20260925 15:04:14.760512 24666 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:14.760598 24666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:14.760864 24666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:14.763604 24666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "3323fc84267c41a8931cc3aa93fe1bfe"
format_stamp: "Formatted at 2026-09-25 15:04:14 on dist-test-slave-nsrc"
server_key: "d835734e0ff719d6c414852d75972970"
server_key_iv: "65b47a43db7ae38dc4643b59e14c26b3"
server_key_version: "encryptionkey@0"
I20260925 15:04:14.764451 24666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "3323fc84267c41a8931cc3aa93fe1bfe"
format_stamp: "Formatted at 2026-09-25 15:04:14 on dist-test-slave-nsrc"
server_key: "d835734e0ff719d6c414852d75972970"
server_key_iv: "65b47a43db7ae38dc4643b59e14c26b3"
server_key_version: "encryptionkey@0"
I20260925 15:04:14.769349 24666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:04:14.772634 24681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:14.773896 24666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:04:14.774041 24666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "3323fc84267c41a8931cc3aa93fe1bfe"
format_stamp: "Formatted at 2026-09-25 15:04:14 on dist-test-slave-nsrc"
server_key: "d835734e0ff719d6c414852d75972970"
server_key_iv: "65b47a43db7ae38dc4643b59e14c26b3"
server_key_version: "encryptionkey@0"
I20260925 15:04:14.774220 24666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:14.789696 24666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:14.793705 24666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:14.793957 24666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:14.796490 24666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:14.796584 24666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:14.796721 24666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:14.796787 24666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:04:14 dist-test-slave-nsrc krb5kdc[24550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348654, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:14.818892 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:14.797445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:34055 (local address 127.22.236.62:33711)
0925 15:04:14.797613 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:14.797618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:14.798961 (+ 1343us) server_negotiation.cc:407] Connection header received
0925 15:04:14.800955 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:14.800961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:14.801043 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:14.801153 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:14.803709 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:14.804898 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:14.807293 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:14.807947 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:14.808092 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:14.812349 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:14.812355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:14.812360 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:14.812399 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:14.815402 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:14.816446 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:14.816452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:14.816455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:14.816602 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:14.817130 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:14.817134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:14.817136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:14.817406 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:14.817509 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:14.818490 (+ 981us) server_negotiation.cc:299] Negotiation successful
0925 15:04:14.818664 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:04:14.820566 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:14.797890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33711 (local address 127.22.236.1:34055)
0925 15:04:14.798786 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:14.798835 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:14.800650 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:14.801540 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:14.801549 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:14.802684 (+ 1135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:14.803530 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:14.803558 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:14.805081 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:14.805087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:14.807113 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:14.807131 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:14.808203 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:14.808218 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:14.808414 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:14.809229 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:14.809264 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:14.812051 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:14.815533 (+ 3482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:14.815544 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:14.815565 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:14.816295 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:14.816733 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:14.816739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:14.816743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:14.817021 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:14.817522 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:14.817531 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:14.818006 (+ 475us) client_negotiation.cc:769] Sending connection context
0925 15:04:14.818366 (+ 360us) client_negotiation.cc:240] Negotiation successful
0925 15:04:14.818950 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":445,"threads_started":1}
I20260925 15:04:14.844990 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:14.831053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:46203 (local address 127.22.236.62:33711)
0925 15:04:14.831262 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:14.831269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:14.831786 (+ 517us) server_negotiation.cc:407] Connection header received
0925 15:04:14.831954 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:14.831958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:14.832022 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:14.832101 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:14.833304 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:14.834067 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:14.836312 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:14.836886 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:14.837010 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:14.839289 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:14.839293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:14.839296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:14.839325 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:14.842267 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:14.843023 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:14.843028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:14.843032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:14.843186 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:14.843585 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:14.843591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:14.843595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:14.843922 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:14.844055 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:14.844457 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 15:04:14.844650 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:04:14.845223 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:14.831010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33711 (local address 127.22.236.1:46203)
0925 15:04:14.831650 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:14.831672 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:14.831835 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:14.832168 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:14.832173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:14.832767 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:14.833146 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:14.833159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:14.834205 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:14.834210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:14.836084 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:14.836104 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:14.837092 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:14.837098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:14.837267 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:14.837911 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:14.837935 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:14.839116 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:14.842403 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:14.842408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:14.842413 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:14.842912 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:14.843310 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:14.843318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:14.843322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:14.843479 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:14.844022 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:14.844027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:14.844226 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 15:04:14.844358 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 15:04:14.844776 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":306,"threads_started":1}
I20260925 15:04:14.846143 24666 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:44881
I20260925 15:04:14.846167 24800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:44881 every 8 connection(s)
I20260925 15:04:14.847373 24666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:04:14.849228 24801 heartbeater.cc:347] Connected to a master server at 127.22.236.62:33711
I20260925 15:04:14.849424 24801 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:14.850010 24801 heartbeater.cc:511] Master 127.22.236.62:33711 requested a full tablet report, sending...
I20260925 15:04:14.851851 24595 ts_manager.cc:194] Registered new tserver with Master: 3323fc84267c41a8931cc3aa93fe1bfe (127.22.236.1:44881)
I20260925 15:04:14.853338 24595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:34055
I20260925 15:04:14.855872 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 24666
I20260925 15:04:14.856042 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:04:14.856632 23472 external_mini_cluster.cc:1428] Setting key f21f596425dd33fcee3eaf075fbd035a
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:04:14.889086 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33711
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:40827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:15.192243 24805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:15.192528 24805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:15.192656 24805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:15.199647 24805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:15.199756 24805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:15.199889 24805 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:04:15.199990 24805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:04:15.209535 24805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:40827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.236.62:33711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:15.210745 24805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:15.212052 24805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:15.224547 24811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:15.224583 24810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:15.224639 24813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:15.225739 24805 server_base.cc:1034] running on GCE node
I20260925 15:04:15.226364 24805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:15.227294 24805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:15.228508 24805 hybrid_clock.cc:648] HybridClock initialized: now 1790348655228440 us; error 45 us; skew 500 ppm
I20260925 15:04:15.228792 24805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:15 dist-test-slave-nsrc krb5kdc[24550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348655, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:15.231709 24805 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:04:15.233143 24805 webserver.cc:460] Webserver started at http://127.22.236.2:35523/ using document root <none> and password file <none>
I20260925 15:04:15.233800 24805 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:15.233886 24805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:15.234138 24805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:15.236999 24805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "a7c5037cbbc145bdb51997721a35701e"
format_stamp: "Formatted at 2026-09-25 15:04:15 on dist-test-slave-nsrc"
server_key: "283bc63e36ceb5f9dc82a07978218156"
server_key_iv: "50c87397f9404ea641caae300cbfbd5f"
server_key_version: "encryptionkey@0"
I20260925 15:04:15.237701 24805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "a7c5037cbbc145bdb51997721a35701e"
format_stamp: "Formatted at 2026-09-25 15:04:15 on dist-test-slave-nsrc"
server_key: "283bc63e36ceb5f9dc82a07978218156"
server_key_iv: "50c87397f9404ea641caae300cbfbd5f"
server_key_version: "encryptionkey@0"
I20260925 15:04:15.242587 24805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:04:15.245905 24820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:15.247202 24805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:04:15.247349 24805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "a7c5037cbbc145bdb51997721a35701e"
format_stamp: "Formatted at 2026-09-25 15:04:15 on dist-test-slave-nsrc"
server_key: "283bc63e36ceb5f9dc82a07978218156"
server_key_iv: "50c87397f9404ea641caae300cbfbd5f"
server_key_version: "encryptionkey@0"
I20260925 15:04:15.247521 24805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:15.277928 24805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:15.282320 24805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:15.282603 24805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:15.285665 24805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:15.285784 24805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:15.285923 24805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:15.285990 24805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:04:15 dist-test-slave-nsrc krb5kdc[24550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348655, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:15.308179 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.285878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:49509 (local address 127.22.236.62:33711)
0925 15:04:15.286070 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:15.286077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:15.287527 (+ 1450us) server_negotiation.cc:407] Connection header received
0925 15:04:15.289597 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:15.289604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:15.289686 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:15.289756 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:15.292323 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.293497 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.296296 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.296762 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.296878 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.301393 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:15.301401 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:15.301405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:15.301450 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:15.304735 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:15.305496 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:15.305501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:15.305505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:15.305657 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:15.306324 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:15.306330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:15.306333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:15.306626 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:15.306774 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:15.307749 (+ 975us) server_negotiation.cc:299] Negotiation successful
0925 15:04:15.307930 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:04:15.309893 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.286571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33711 (local address 127.22.236.2:49509)
0925 15:04:15.287375 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:15.287424 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:15.289294 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:15.290212 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:15.290227 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:15.290988 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:15.292061 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:15.292091 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.293707 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.293713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:15.296120 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:15.296155 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.297168 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.297176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:15.297472 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.298212 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:15.298233 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:15.301072 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:15.304877 (+ 3805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:15.304889 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:15.304905 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:15.305351 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:15.305800 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:15.305807 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:15.305811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:15.306161 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:15.306782 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:15.306791 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:15.307342 (+ 551us) client_negotiation.cc:769] Sending connection context
0925 15:04:15.307660 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 15:04:15.308290 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":347,"threads_started":1}
I20260925 15:04:15.330351 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.316578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:41257 (local address 127.22.236.62:33711)
0925 15:04:15.316901 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:15.316905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:15.317991 (+ 1086us) server_negotiation.cc:407] Connection header received
0925 15:04:15.318156 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:15.318160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:15.318218 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:15.318295 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:15.319431 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.320133 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.322024 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.322742 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.322864 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.324797 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:15.324804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:15.324808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:15.324845 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:15.327828 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:15.328547 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:15.328553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:15.328557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:15.328669 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:15.328997 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:15.329002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:15.329005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:15.329297 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:15.329420 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:15.329804 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 15:04:15.330009 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260925 15:04:15.330439 24905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.316650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33711 (local address 127.22.236.2:41257)
0925 15:04:15.317880 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:15.317903 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:15.318054 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:15.318341 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:15.318346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:15.318917 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:15.319309 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:15.319322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.320310 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.320313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:15.321878 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:15.321892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.322918 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.322923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:15.323095 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.323699 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:15.323716 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:15.324562 (+ 846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:15.327954 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:15.327960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:15.327964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:15.328439 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:15.328770 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:15.328775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:15.328779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:15.328900 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:15.329413 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:15.329417 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:15.329584 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 15:04:15.329700 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 15:04:15.330003 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"thread_start_us":883,"threads_started":1}
I20260925 15:04:15.335505 24805 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:41847
I20260925 15:04:15.335533 24939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:41847 every 8 connection(s)
I20260925 15:04:15.336751 24805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:04:15.338536 24940 heartbeater.cc:347] Connected to a master server at 127.22.236.62:33711
I20260925 15:04:15.338737 24940 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:15.339330 24940 heartbeater.cc:511] Master 127.22.236.62:33711 requested a full tablet report, sending...
I20260925 15:04:15.340595 24595 ts_manager.cc:194] Registered new tserver with Master: a7c5037cbbc145bdb51997721a35701e (127.22.236.2:41847)
I20260925 15:04:15.341411 24595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:49509
I20260925 15:04:15.344259 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 24805
I20260925 15:04:15.344434 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:04:15.344846 23472 external_mini_cluster.cc:1428] Setting key 0211ec141ce49fd3f6a88a53520bab7c
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:04:15.378870 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33711
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:40827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:15.683388 24944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:15.683661 24944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:15.683784 24944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:15.690971 24944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:15.691080 24944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:15.691188 24944 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:04:15.691293 24944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:04:15.700927 24944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:40827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.236.62:33711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:15.702173 24944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:15.703447 24944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:15.715863 24949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:15.715837 24950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:15.715898 24952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:15.716457 24944 server_base.cc:1034] running on GCE node
I20260925 15:04:15.717336 24944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:15.718233 24944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:15.719448 24944 hybrid_clock.cc:648] HybridClock initialized: now 1790348655719381 us; error 43 us; skew 500 ppm
I20260925 15:04:15.719722 24944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:15 dist-test-slave-nsrc krb5kdc[24550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348655, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:15.722810 24944 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:04:15.724224 24944 webserver.cc:460] Webserver started at http://127.22.236.3:36547/ using document root <none> and password file <none>
I20260925 15:04:15.724892 24944 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:15.724982 24944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:15.725274 24944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:15.728050 24944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "a35825b136bc4e28b16ac0860a8e99a3"
format_stamp: "Formatted at 2026-09-25 15:04:15 on dist-test-slave-nsrc"
server_key: "2219165b6cfc00d74d59aa71adecef8a"
server_key_iv: "0d7266f59c67fc1700969dc6ecb63df2"
server_key_version: "encryptionkey@0"
I20260925 15:04:15.728947 24944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "a35825b136bc4e28b16ac0860a8e99a3"
format_stamp: "Formatted at 2026-09-25 15:04:15 on dist-test-slave-nsrc"
server_key: "2219165b6cfc00d74d59aa71adecef8a"
server_key_iv: "0d7266f59c67fc1700969dc6ecb63df2"
server_key_version: "encryptionkey@0"
I20260925 15:04:15.733593 24944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:04:15.736913 24959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:15.738134 24944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:04:15.738257 24944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "a35825b136bc4e28b16ac0860a8e99a3"
format_stamp: "Formatted at 2026-09-25 15:04:15 on dist-test-slave-nsrc"
server_key: "2219165b6cfc00d74d59aa71adecef8a"
server_key_iv: "0d7266f59c67fc1700969dc6ecb63df2"
server_key_version: "encryptionkey@0"
I20260925 15:04:15.738431 24944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:15.750269 24944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:15.754078 24944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:15.754302 24944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:15.756631 24944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:15.756703 24944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:15.756829 24944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:15.756883 24944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:04:15 dist-test-slave-nsrc krb5kdc[24550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348655, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:15.779081 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.757279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:39059 (local address 127.22.236.62:33711)
0925 15:04:15.757473 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:15.757479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:15.758757 (+ 1278us) server_negotiation.cc:407] Connection header received
0925 15:04:15.761066 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:15.761073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:15.761148 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:15.761204 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:15.762966 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.764071 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.766416 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.767026 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.767148 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.771790 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:15.771797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:15.771801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:15.771852 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:15.775402 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:15.776246 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:15.776251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:15.776264 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:15.776451 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:15.776905 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:15.776911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:15.776914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:15.777176 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:15.777316 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:15.778482 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0925 15:04:15.778707 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 15:04:15.780561 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.757775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33711 (local address 127.22.236.3:39059)
0925 15:04:15.758576 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:15.758623 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:15.760672 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:15.761724 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:15.761735 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:15.762261 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:15.762788 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:15.762804 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.764257 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.764262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:15.766272 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:15.766288 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.767239 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.767246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:15.767574 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.768278 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:15.768309 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:15.771483 (+ 3174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:15.775529 (+ 4046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:15.775542 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:15.775561 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:15.776106 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:15.776564 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:15.776567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:15.776570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:15.776775 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:15.777409 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:15.777418 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:15.777835 (+ 417us) client_negotiation.cc:769] Sending connection context
0925 15:04:15.778164 (+ 329us) client_negotiation.cc:240] Negotiation successful
0925 15:04:15.778746 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":358,"threads_started":1}
I20260925 15:04:15.793108 24944 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:38931
I20260925 15:04:15.793139 25078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:38931 every 8 connection(s)
I20260925 15:04:15.794422 24944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:04:15.796366 25079 heartbeater.cc:347] Connected to a master server at 127.22.236.62:33711
I20260925 15:04:15.796562 25079 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:15.797158 25079 heartbeater.cc:511] Master 127.22.236.62:33711 requested a full tablet report, sending...
I20260925 15:04:15.798498 24595 ts_manager.cc:194] Registered new tserver with Master: a35825b136bc4e28b16ac0860a8e99a3 (127.22.236.3:38931)
I20260925 15:04:15.799477 24595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:39059
I20260925 15:04:15.800182 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.787199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:51023 (local address 127.22.236.62:33711)
0925 15:04:15.787380 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:15.787386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:15.788210 (+ 824us) server_negotiation.cc:407] Connection header received
0925 15:04:15.788409 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:15.788414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:15.788494 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:15.788597 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:15.789867 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.790906 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.792184 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.792800 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.792927 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.794712 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:15.794719 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:15.794722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:15.794750 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:15.797796 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:15.798442 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:15.798446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:15.798448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:15.798609 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:15.798952 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:15.798955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:15.798957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:15.799192 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:15.799294 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:15.799719 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 15:04:15.799920 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:04:15.800365 25067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.787294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33711 (local address 127.22.236.3:51023)
0925 15:04:15.788078 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:15.788097 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:15.788275 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:15.788653 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:15.788658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:15.789269 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:15.789700 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:15.789717 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.791042 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.791048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:15.792035 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:15.792047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.793000 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.793010 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:15.793221 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.793819 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:15.793835 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:15.794542 (+ 707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:15.797905 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:15.797910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:15.797914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:15.798336 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:15.798704 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:15.798708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:15.798711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:15.798853 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:15.799294 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:15.799299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:15.799479 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 15:04:15.799622 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 15:04:15.799914 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":531,"threads_started":1}
I20260925 15:04:15.802852 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 24944
I20260925 15:04:15.803004 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:04:15.803428 23472 external_mini_cluster.cc:1428] Setting key 08333c7146d62afd6773805b87c6c5a0
I20260925 15:04:15.806008 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:04:15 dist-test-slave-nsrc krb5kdc[24550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:04:15 dist-test-slave-nsrc krb5kdc[24550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:15.846333 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.833149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.22.236.62:33711)
0925 15:04:15.833327 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:15.833332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:15.833672 (+ 340us) server_negotiation.cc:407] Connection header received
0925 15:04:15.833825 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:15.833829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:15.833891 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:15.833976 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:15.835268 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.836064 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.837366 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.837781 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.837880 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.841499 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:15.841504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:15.841507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:15.841544 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:15.843912 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:15.844564 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:15.844569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:15.844571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:15.844706 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:15.845064 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:15.845068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:15.845070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:15.845336 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:15.845542 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:15.845856 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 15:04:15.846055 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:04:15.850418 24595 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 15:04:15.856106 24801 heartbeater.cc:503] Master 127.22.236.62:33711 was elected leader, sending a full tablet report...
W20260925 15:04:15.857193 24595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:04:15.878746 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.868830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:38931 (local address 127.0.0.1:57226)
0925 15:04:15.869606 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:15.869666 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:15.869829 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:15.870925 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:15.870933 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:15.870960 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:15.871355 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:15.871369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.873342 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.873348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:15.875979 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:15.875993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.877477 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.877483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:15.877660 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.877804 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 15:04:15.878105 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 15:04:15.878291 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":51,"spinlock_wait_cycles":12032,"thread_start_us":372,"threads_started":1}
I20260925 15:04:15.879137 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.868369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:44881 (local address 127.0.0.1:53012)
0925 15:04:15.869606 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:15.869666 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:15.869829 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:15.873049 (+ 3220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:15.873053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:15.873075 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:15.873421 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:15.873430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.874724 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.874729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:15.876466 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:15.876478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.878313 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.878319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:15.878469 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.878502 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:04:15.878562 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:04:15.878676 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":3,"spinlock_wait_cycles":384,"thread_start_us":361,"threads_started":1}
I20260925 15:04:15.879446 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.869221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57226 (local address 127.22.236.3:38931)
0925 15:04:15.870383 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:15.870391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:15.870421 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:04:15.870522 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:15.870529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:15.870733 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:15.870905 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:15.871586 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.873204 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.876157 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.877310 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.877425 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.877870 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:15.878987 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0925 15:04:15.879143 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":990,"thread_start_us":845,"threads_started":1}
I20260925 15:04:15.879779 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.868520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.22.236.1:44881)
0925 15:04:15.872599 (+ 4079us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:15.872605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:15.872626 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:04:15.872698 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:15.872702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:15.872852 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:15.873064 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:15.873582 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.874615 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.876623 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.878099 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.879048 (+ 949us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.879328 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:15.879399 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 15:04:15.879540 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3950,"thread_start_us":3809,"threads_started":1}
I20260925 15:04:15.880021 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.869622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41847 (local address 127.0.0.1:33390)
0925 15:04:15.871524 (+ 1902us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:15.871542 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:15.871639 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:15.872475 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:15.872479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:15.872501 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:15.872828 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:15.872839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.874507 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.874512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:15.877097 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:15.877110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.879380 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.879387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:15.879523 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.879553 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:04:15.879622 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:04:15.879740 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1789,"mutex_wait_us":662,"thread_start_us":1660,"threads_started":1}
I20260925 15:04:15.880716 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:15.870074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.22.236.2:41847)
0925 15:04:15.870559 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:15.870567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:15.871747 (+ 1180us) server_negotiation.cc:407] Connection header received
0925 15:04:15.871855 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:15.871862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:15.872087 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:15.872302 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:15.872992 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.874384 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.877267 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:15.878549 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:15.878667 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:15.880268 (+ 1601us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:15.880335 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 15:04:15.880465 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":227,"threads_started":1}
I20260925 15:04:15.883095 25009 tablet_service.cc:1467] Processing CreateTablet for tablet a7fbd4e758a542b8a36e8bc474c7d67b (DEFAULT_TABLE table=test-table [id=af553faca83842a4ade4934d494d46d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:15.883091 24730 tablet_service.cc:1467] Processing CreateTablet for tablet a7fbd4e758a542b8a36e8bc474c7d67b (DEFAULT_TABLE table=test-table [id=af553faca83842a4ade4934d494d46d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:15.883572 24870 tablet_service.cc:1467] Processing CreateTablet for tablet a7fbd4e758a542b8a36e8bc474c7d67b (DEFAULT_TABLE table=test-table [id=af553faca83842a4ade4934d494d46d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:15.884244 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fbd4e758a542b8a36e8bc474c7d67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:15.884420 24730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fbd4e758a542b8a36e8bc474c7d67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:15.884744 24870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fbd4e758a542b8a36e8bc474c7d67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:15.894754 25098 tablet_bootstrap.cc:492] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3: Bootstrap starting.
I20260925 15:04:15.897753 25099 tablet_bootstrap.cc:492] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e: Bootstrap starting.
I20260925 15:04:15.898762 25098 tablet_bootstrap.cc:654] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:15.899986 25098 log.cc:824] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:15.901046 25099 tablet_bootstrap.cc:654] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:15.902288 25099 log.cc:824] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:15.903895 25100 tablet_bootstrap.cc:492] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe: Bootstrap starting.
I20260925 15:04:15.905625 25099 tablet_bootstrap.cc:492] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e: No bootstrap required, opened a new log
I20260925 15:04:15.905877 25099 ts_tablet_manager.cc:1397] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 15:04:15.907280 25100 tablet_bootstrap.cc:654] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:15.908284 25100 log.cc:824] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:15.908563 25098 tablet_bootstrap.cc:492] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3: No bootstrap required, opened a new log
I20260925 15:04:15.908805 25098 ts_tablet_manager.cc:1397] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20260925 15:04:15.912437 25100 tablet_bootstrap.cc:492] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe: No bootstrap required, opened a new log
I20260925 15:04:15.912678 25100 ts_tablet_manager.cc:1397] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 15:04:15.915515 25099 raft_consensus.cc:348] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:15.915930 25099 raft_consensus.cc:374] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:15.916021 25099 raft_consensus.cc:729] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c5037cbbc145bdb51997721a35701e, State: Initialized, Role: FOLLOWER
I20260925 15:04:15.916631 25099 consensus_queue.cc:260] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:15.917663 24940 heartbeater.cc:503] Master 127.22.236.62:33711 was elected leader, sending a full tablet report...
I20260925 15:04:15.918259 25099 ts_tablet_manager.cc:1428] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260925 15:04:15.918902 25098 raft_consensus.cc:348] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:15.919252 25098 raft_consensus.cc:374] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:15.919358 25098 raft_consensus.cc:729] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35825b136bc4e28b16ac0860a8e99a3, State: Initialized, Role: FOLLOWER
I20260925 15:04:15.919991 25098 consensus_queue.cc:260] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:15.920614 25100 raft_consensus.cc:348] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:15.920920 25100 raft_consensus.cc:374] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:15.920989 25100 raft_consensus.cc:729] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3323fc84267c41a8931cc3aa93fe1bfe, State: Initialized, Role: FOLLOWER
I20260925 15:04:15.921450 25100 consensus_queue.cc:260] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:15.924849 25100 ts_tablet_manager.cc:1428] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260925 15:04:15.927364 25079 heartbeater.cc:503] Master 127.22.236.62:33711 was elected leader, sending a full tablet report...
I20260925 15:04:15.928256 25098 ts_tablet_manager.cc:1428] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3: Time spent starting tablet: real 0.019s user 0.014s sys 0.005s
W20260925 15:04:16.046201 25080 tablet.cc:2369] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:04:16.088581 24941 tablet.cc:2369] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:04:16.099731 24802 tablet.cc:2369] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:04:16.276474 25106 raft_consensus.cc:482] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:04:16.276800 25106 raft_consensus.cc:504] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.278489 25106 leader_election.cc:288] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7c5037cbbc145bdb51997721a35701e (127.22.236.2:41847), a35825b136bc4e28b16ac0860a8e99a3 (127.22.236.3:38931)
I20260925 15:04:16.285612 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:16.278943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:43067 (local address 127.22.236.3:38931)
0925 15:04:16.279127 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:16.279132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:16.279478 (+ 346us) server_negotiation.cc:407] Connection header received
0925 15:04:16.279682 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:16.279687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:16.279744 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:16.279814 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:16.280348 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.281266 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.283865 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.284733 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.284884 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:16.284957 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:16.285245 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 15:04:16.285373 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:04:16.285923 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:16.279022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:53721 (local address 127.22.236.2:41847)
0925 15:04:16.279227 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:16.279232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:16.279846 (+ 614us) server_negotiation.cc:407] Connection header received
0925 15:04:16.280020 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:16.280024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:16.280082 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:16.280175 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:16.280692 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.281463 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.283843 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.285083 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.285246 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:16.285322 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:16.285581 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 15:04:16.285695 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:04:16.286093 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:16.278710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41847 (local address 127.22.236.1:53721)
0925 15:04:16.279736 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:16.279750 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:16.279877 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:16.280207 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:16.280211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:16.280232 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:16.280542 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:16.280551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.281620 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.281625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:16.283676 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:16.283694 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.285274 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.285279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:16.285437 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:16.285471 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:04:16.285557 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:04:16.285676 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":436,"thread_start_us":300,"threads_started":1}
I20260925 15:04:16.286229 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:16.278710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:38931 (local address 127.22.236.1:43067)
0925 15:04:16.279369 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:16.279384 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:16.279539 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:16.279852 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:16.279856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:16.279881 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:16.280222 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:16.280232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.281419 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.281424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:16.283695 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:16.283704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.284929 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.284936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:16.285081 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:16.285105 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 15:04:16.285774 (+ 669us) client_negotiation.cc:240] Negotiation successful
0925 15:04:16.285904 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":32,"thread_start_us":319,"threads_started":1}
I20260925 15:04:16.287076 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fbd4e758a542b8a36e8bc474c7d67b" candidate_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35825b136bc4e28b16ac0860a8e99a3" is_pre_election: true
I20260925 15:04:16.287186 24892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fbd4e758a542b8a36e8bc474c7d67b" candidate_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c5037cbbc145bdb51997721a35701e" is_pre_election: true
I20260925 15:04:16.287577 25029 raft_consensus.cc:2454] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3323fc84267c41a8931cc3aa93fe1bfe in term 0.
I20260925 15:04:16.287642 24892 raft_consensus.cc:2454] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3323fc84267c41a8931cc3aa93fe1bfe in term 0.
I20260925 15:04:16.288192 24683 leader_election.cc:302] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3323fc84267c41a8931cc3aa93fe1bfe, a35825b136bc4e28b16ac0860a8e99a3; no voters:
I20260925 15:04:16.288545 25106 raft_consensus.cc:2781] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:04:16.288694 25106 raft_consensus.cc:482] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:04:16.288777 25106 raft_consensus.cc:3037] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:16.290814 25106 raft_consensus.cc:504] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.291363 25106 leader_election.cc:288] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [CANDIDATE]: Term 1 election: Requested vote from peers a7c5037cbbc145bdb51997721a35701e (127.22.236.2:41847), a35825b136bc4e28b16ac0860a8e99a3 (127.22.236.3:38931)
I20260925 15:04:16.291829 24892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fbd4e758a542b8a36e8bc474c7d67b" candidate_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c5037cbbc145bdb51997721a35701e"
I20260925 15:04:16.291829 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fbd4e758a542b8a36e8bc474c7d67b" candidate_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35825b136bc4e28b16ac0860a8e99a3"
I20260925 15:04:16.292037 25029 raft_consensus.cc:3037] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:16.292037 24892 raft_consensus.cc:3037] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:16.295099 24892 raft_consensus.cc:2454] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3323fc84267c41a8931cc3aa93fe1bfe in term 1.
I20260925 15:04:16.295446 25029 raft_consensus.cc:2454] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3323fc84267c41a8931cc3aa93fe1bfe in term 1.
I20260925 15:04:16.295498 24682 leader_election.cc:302] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3323fc84267c41a8931cc3aa93fe1bfe, a7c5037cbbc145bdb51997721a35701e; no voters:
I20260925 15:04:16.295811 25106 raft_consensus.cc:2781] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:16.296270 25106 raft_consensus.cc:686] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [term 1 LEADER]: Becoming Leader. State: Replica: 3323fc84267c41a8931cc3aa93fe1bfe, State: Running, Role: LEADER
I20260925 15:04:16.296746 25106 consensus_queue.cc:237] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.300827 24594 catalog_manager.cc:5495] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe reported cstate change: term changed from 0 to 1, leader changed from <none> to 3323fc84267c41a8931cc3aa93fe1bfe (127.22.236.1). New cstate: current_term: 1 leader_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } health_report { overall_health: UNKNOWN } } }
I20260925 15:04:16.339109 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:16.333352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.22.236.1:44881)
0925 15:04:16.333539 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:16.333545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:16.333650 (+ 105us) server_negotiation.cc:407] Connection header received
0925 15:04:16.333952 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:16.333958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:16.334052 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:16.334169 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:16.334527 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.335657 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.337074 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.337474 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.337578 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:16.338049 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:16.338530 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:16.338653 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:16.338782 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 15:04:16.338861 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:04:16.347651 24892 raft_consensus.cc:1261] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e [term 1 FOLLOWER]: Refusing update from remote peer 3323fc84267c41a8931cc3aa93fe1bfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:16.347846 25029 raft_consensus.cc:1261] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3 [term 1 FOLLOWER]: Refusing update from remote peer 3323fc84267c41a8931cc3aa93fe1bfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:16.348701 25106 consensus_queue.cc:1035] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:16.348979 25109 consensus_queue.cc:1035] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:16.363790 25112 mvcc.cc:205] Tried to move back new op lower bound from 7333268096390643712 to 7333268096194752512. Current Snapshot: MvccSnapshot[applied={T|T < 7333268096390643712}]
I20260925 15:04:16.367204 25115 mvcc.cc:205] Tried to move back new op lower bound from 7333268096390643712 to 7333268096194752512. Current Snapshot: MvccSnapshot[applied={T|T < 7333268096390643712}]
I20260925 15:04:16.381271 24595 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:16.381501 24595 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:16.384918 24595 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654: Sending DeleteTablet for 3 replicas of tablet a7fbd4e758a542b8a36e8bc474c7d67b
I20260925 15:04:16.386242 25009 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fbd4e758a542b8a36e8bc474c7d67b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:16 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:57226
I20260925 15:04:16.386464 24870 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fbd4e758a542b8a36e8bc474c7d67b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:16 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:33390
I20260925 15:04:16.387169 24730 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fbd4e758a542b8a36e8bc474c7d67b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:16 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:53012
I20260925 15:04:16.387493 25122 tablet_replica.cc:330] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e: stopping tablet replica
I20260925 15:04:16.387995 25122 raft_consensus.cc:2229] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:16.387905 24595 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:04:16.388687 24595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:04:16.388865 25122 raft_consensus.cc:2258] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:16.391098 25122 ts_tablet_manager.cc:1905] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:16.396082 24731 tablet_service.cc:1467] Processing CreateTablet for tablet cfead76f9b9047e2a9517a90408b853f (DEFAULT_TABLE table=test-table [id=142b1676259b46ce9b723af9c23f7ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:16.396255 25008 tablet_service.cc:1467] Processing CreateTablet for tablet cfead76f9b9047e2a9517a90408b853f (DEFAULT_TABLE table=test-table [id=142b1676259b46ce9b723af9c23f7ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:16.396667 24731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfead76f9b9047e2a9517a90408b853f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:16.396703 25008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfead76f9b9047e2a9517a90408b853f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:16.400738 24870 tablet_service.cc:1467] Processing CreateTablet for tablet cfead76f9b9047e2a9517a90408b853f (DEFAULT_TABLE table=test-table [id=142b1676259b46ce9b723af9c23f7ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:16.401149 24870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfead76f9b9047e2a9517a90408b853f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:16.402539 25098 tablet_bootstrap.cc:492] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3: Bootstrap starting.
I20260925 15:04:16.404768 25121 tablet_replica.cc:330] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3: stopping tablet replica
I20260925 15:04:16.404824 25098 tablet_bootstrap.cc:654] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:16.405225 25121 raft_consensus.cc:2229] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:16.405717 25121 raft_consensus.cc:2258] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:16.407135 25098 tablet_bootstrap.cc:492] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3: No bootstrap required, opened a new log
I20260925 15:04:16.407265 25100 tablet_bootstrap.cc:492] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe: Bootstrap starting.
I20260925 15:04:16.407266 25098 ts_tablet_manager.cc:1397] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260925 15:04:16.407951 25098 raft_consensus.cc:348] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.408202 25098 raft_consensus.cc:374] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:16.408294 25098 raft_consensus.cc:729] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35825b136bc4e28b16ac0860a8e99a3, State: Initialized, Role: FOLLOWER
I20260925 15:04:16.408390 25099 tablet_bootstrap.cc:492] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e: Bootstrap starting.
I20260925 15:04:16.408576 25098 consensus_queue.cc:260] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.408798 25100 tablet_bootstrap.cc:654] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:16.409181 25098 ts_tablet_manager.cc:1428] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 15:04:16.409982 25099 tablet_bootstrap.cc:654] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:16.413257 25122 ts_tablet_manager.cc:1918] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:04:16.413429 25122 log.cc:1196] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/a7fbd4e758a542b8a36e8bc474c7d67b
I20260925 15:04:16.413960 25122 ts_tablet_manager.cc:1939] T a7fbd4e758a542b8a36e8bc474c7d67b P a7c5037cbbc145bdb51997721a35701e: Deleting consensus metadata
I20260925 15:04:16.414458 25123 tablet_replica.cc:330] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe: stopping tablet replica
I20260925 15:04:16.415031 25123 raft_consensus.cc:2229] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:04:16.415755 24579 catalog_manager.cc:4841] TS a7c5037cbbc145bdb51997721a35701e (127.22.236.2:41847): tablet a7fbd4e758a542b8a36e8bc474c7d67b (table test-table [id=af553faca83842a4ade4934d494d46d8]) successfully deleted
I20260925 15:04:16.416556 25123 raft_consensus.cc:2258] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:16.417323 25100 tablet_bootstrap.cc:492] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe: No bootstrap required, opened a new log
I20260925 15:04:16.417485 25100 ts_tablet_manager.cc:1397] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20260925 15:04:16.418295 25100 raft_consensus.cc:348] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.418531 25100 raft_consensus.cc:374] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:16.418581 25099 tablet_bootstrap.cc:492] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e: No bootstrap required, opened a new log
I20260925 15:04:16.418627 25100 raft_consensus.cc:729] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3323fc84267c41a8931cc3aa93fe1bfe, State: Initialized, Role: FOLLOWER
I20260925 15:04:16.418706 25099 ts_tablet_manager.cc:1397] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260925 15:04:16.419452 25099 raft_consensus.cc:348] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.419665 25099 raft_consensus.cc:374] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:16.419759 25099 raft_consensus.cc:729] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c5037cbbc145bdb51997721a35701e, State: Initialized, Role: FOLLOWER
I20260925 15:04:16.420100 25123 ts_tablet_manager.cc:1905] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:16.420029 25099 consensus_queue.cc:260] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.420624 25121 ts_tablet_manager.cc:1905] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:16.421026 25100 consensus_queue.cc:260] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.422498 25100 ts_tablet_manager.cc:1428] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260925 15:04:16.424175 25099 ts_tablet_manager.cc:1428] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260925 15:04:16.427838 25121 ts_tablet_manager.cc:1918] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:04:16.428066 25121 log.cc:1196] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/a7fbd4e758a542b8a36e8bc474c7d67b
I20260925 15:04:16.428758 25121 ts_tablet_manager.cc:1939] T a7fbd4e758a542b8a36e8bc474c7d67b P a35825b136bc4e28b16ac0860a8e99a3: Deleting consensus metadata
I20260925 15:04:16.429603 25123 ts_tablet_manager.cc:1918] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:04:16.429764 25123 log.cc:1196] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/a7fbd4e758a542b8a36e8bc474c7d67b
I20260925 15:04:16.430264 25123 ts_tablet_manager.cc:1939] T a7fbd4e758a542b8a36e8bc474c7d67b P 3323fc84267c41a8931cc3aa93fe1bfe: Deleting consensus metadata
I20260925 15:04:16.430466 24580 catalog_manager.cc:4841] TS a35825b136bc4e28b16ac0860a8e99a3 (127.22.236.3:38931): tablet a7fbd4e758a542b8a36e8bc474c7d67b (table test-table [id=af553faca83842a4ade4934d494d46d8]) successfully deleted
I20260925 15:04:16.431881 24579 catalog_manager.cc:4841] TS 3323fc84267c41a8931cc3aa93fe1bfe (127.22.236.1:44881): tablet a7fbd4e758a542b8a36e8bc474c7d67b (table test-table [id=af553faca83842a4ade4934d494d46d8]) successfully deleted
I20260925 15:04:16.600802 25104 raft_consensus.cc:482] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:04:16.601079 25104 raft_consensus.cc:504] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.602970 25104 leader_election.cc:288] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3323fc84267c41a8931cc3aa93fe1bfe (127.22.236.1:44881), a35825b136bc4e28b16ac0860a8e99a3 (127.22.236.3:38931)
I20260925 15:04:16.611094 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:16.603593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:37901 (local address 127.22.236.3:38931)
0925 15:04:16.603757 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:16.603763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:16.604434 (+ 671us) server_negotiation.cc:407] Connection header received
0925 15:04:16.604743 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:16.604747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:16.604825 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:16.604907 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:16.605647 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.606580 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.608668 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.609588 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.609793 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:16.609867 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:16.610656 (+ 789us) server_negotiation.cc:299] Negotiation successful
0925 15:04:16.610810 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:04:16.611436 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:16.603138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:44881 (local address 127.22.236.2:54323)
0925 15:04:16.604710 (+ 1572us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:16.604730 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:16.604873 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:16.605204 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:16.605213 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:16.605236 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:16.605596 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:16.605609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.607060 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.607066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:16.609343 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:16.609358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.610694 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.610709 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:16.610853 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:16.610875 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:04:16.610953 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:04:16.611070 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1440,"mutex_wait_us":156,"thread_start_us":627,"threads_started":1}
I20260925 15:04:16.611428 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:16.603346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:54323 (local address 127.22.236.1:44881)
0925 15:04:16.603494 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:16.603499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:16.604837 (+ 1338us) server_negotiation.cc:407] Connection header received
0925 15:04:16.604994 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:16.604999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:16.605083 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:16.605160 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:16.605770 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.606908 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.609523 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.610497 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.610638 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:16.610719 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:16.610996 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 15:04:16.611119 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:04:16.612349 24752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfead76f9b9047e2a9517a90408b853f" candidate_uuid: "a7c5037cbbc145bdb51997721a35701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" is_pre_election: true
I20260925 15:04:16.612602 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:16.603138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:38931 (local address 127.22.236.2:37901)
0925 15:04:16.604327 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:16.604349 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:16.604556 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:16.605020 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:16.605025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:16.605046 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:16.605450 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:16.605463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.606720 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.606726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:16.608521 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:16.608533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.610294 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.610301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:16.610469 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:16.610504 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:04:16.611972 (+ 1468us) client_negotiation.cc:240] Negotiation successful
0925 15:04:16.612103 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":1,"thread_start_us":1257,"threads_started":1}
I20260925 15:04:16.612771 24752 raft_consensus.cc:2454] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7c5037cbbc145bdb51997721a35701e in term 0.
I20260925 15:04:16.613293 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfead76f9b9047e2a9517a90408b853f" candidate_uuid: "a7c5037cbbc145bdb51997721a35701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35825b136bc4e28b16ac0860a8e99a3" is_pre_election: true
I20260925 15:04:16.613392 24821 leader_election.cc:302] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3323fc84267c41a8931cc3aa93fe1bfe, a7c5037cbbc145bdb51997721a35701e; no voters:
I20260925 15:04:16.613533 25029 raft_consensus.cc:2454] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7c5037cbbc145bdb51997721a35701e in term 0.
I20260925 15:04:16.613785 25104 raft_consensus.cc:2781] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:04:16.613907 25104 raft_consensus.cc:482] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:04:16.614003 25104 raft_consensus.cc:3037] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:16.616230 25104 raft_consensus.cc:504] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.616865 25104 leader_election.cc:288] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [CANDIDATE]: Term 1 election: Requested vote from peers 3323fc84267c41a8931cc3aa93fe1bfe (127.22.236.1:44881), a35825b136bc4e28b16ac0860a8e99a3 (127.22.236.3:38931)
I20260925 15:04:16.617676 24752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfead76f9b9047e2a9517a90408b853f" candidate_uuid: "a7c5037cbbc145bdb51997721a35701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3323fc84267c41a8931cc3aa93fe1bfe"
I20260925 15:04:16.617867 24752 raft_consensus.cc:3037] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:16.617837 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfead76f9b9047e2a9517a90408b853f" candidate_uuid: "a7c5037cbbc145bdb51997721a35701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35825b136bc4e28b16ac0860a8e99a3"
I20260925 15:04:16.618031 25029 raft_consensus.cc:3037] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:16.620342 24752 raft_consensus.cc:2454] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c5037cbbc145bdb51997721a35701e in term 1.
I20260925 15:04:16.620692 24821 leader_election.cc:302] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3323fc84267c41a8931cc3aa93fe1bfe, a7c5037cbbc145bdb51997721a35701e; no voters:
I20260925 15:04:16.620940 25104 raft_consensus.cc:2781] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:16.620930 25029 raft_consensus.cc:2454] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c5037cbbc145bdb51997721a35701e in term 1.
I20260925 15:04:16.621681 25104 raft_consensus.cc:686] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [term 1 LEADER]: Becoming Leader. State: Replica: a7c5037cbbc145bdb51997721a35701e, State: Running, Role: LEADER
I20260925 15:04:16.622082 25104 consensus_queue.cc:237] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.626219 24594 catalog_manager.cc:5495] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e reported cstate change: term changed from 0 to 1, leader changed from <none> to a7c5037cbbc145bdb51997721a35701e (127.22.236.2). New cstate: current_term: 1 leader_uuid: "a7c5037cbbc145bdb51997721a35701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:04:16 dist-test-slave-nsrc krb5kdc[24550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348654, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:16.691807 25136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:16.678484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33711 (local address 127.0.0.1:39830)
0925 15:04:16.679077 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:16.679104 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:16.679275 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:16.679616 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:16.679620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:16.680329 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:16.680525 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:16.680534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.681948 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.681953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:16.682920 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:16.682932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.683584 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.683589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:16.683765 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:16.684428 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:16.684446 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:16.686756 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:16.689490 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:16.689503 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:16.689512 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:16.689886 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:16.690196 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:16.690201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:16.690205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:16.690350 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:16.690812 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:16.690820 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:16.690970 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 15:04:16.691111 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 15:04:16.691386 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":320,"threads_started":1}
I20260925 15:04:16.691884 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:16.678516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.22.236.62:33711)
0925 15:04:16.679375 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:16.679380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:16.679398 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:04:16.679435 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:16.679438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:16.679489 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:16.679569 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:16.680673 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.681826 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.683043 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.683433 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.683558 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:16.686944 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:16.686949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:16.686952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:16.686978 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:16.689374 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:16.689986 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:16.689991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:16.689994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:16.690103 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:16.690442 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:16.690447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:16.690450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:16.690684 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:16.690836 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:16.691200 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 15:04:16.691513 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":748,"thread_start_us":624,"threads_started":1}
I20260925 15:04:16.696566 24594 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:39830:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:04:16.697606 24594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:04:16.704003 24870 tablet_service.cc:1467] Processing CreateTablet for tablet 92d4ff44473e44eba414e62f6f833bba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a35077b69ad43228f5523148dc5097f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:04:16.704002 24730 tablet_service.cc:1467] Processing CreateTablet for tablet 92d4ff44473e44eba414e62f6f833bba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a35077b69ad43228f5523148dc5097f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:04:16.704344 25009 tablet_service.cc:1467] Processing CreateTablet for tablet 92d4ff44473e44eba414e62f6f833bba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a35077b69ad43228f5523148dc5097f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:04:16.704568 24730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d4ff44473e44eba414e62f6f833bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:16.704566 24870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d4ff44473e44eba414e62f6f833bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:16.704877 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d4ff44473e44eba414e62f6f833bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:16.711323 25100 tablet_bootstrap.cc:492] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe: Bootstrap starting.
I20260925 15:04:16.711944 25098 tablet_bootstrap.cc:492] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3: Bootstrap starting.
I20260925 15:04:16.712399 25099 tablet_bootstrap.cc:492] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e: Bootstrap starting.
I20260925 15:04:16.713063 25100 tablet_bootstrap.cc:654] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:16.713578 25098 tablet_bootstrap.cc:654] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:16.714355 25099 tablet_bootstrap.cc:654] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:16.723166 25099 tablet_bootstrap.cc:492] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e: No bootstrap required, opened a new log
I20260925 15:04:16.723363 25099 ts_tablet_manager.cc:1397] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260925 15:04:16.724179 25099 raft_consensus.cc:348] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.724422 25099 raft_consensus.cc:374] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:16.724498 25099 raft_consensus.cc:729] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c5037cbbc145bdb51997721a35701e, State: Initialized, Role: FOLLOWER
I20260925 15:04:16.724754 25099 consensus_queue.cc:260] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.725310 25104 tablet_replica.cc:439] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } } }
I20260925 15:04:16.725603 25104 tablet_replica.cc:442] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:04:16.726002 25100 tablet_bootstrap.cc:492] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe: No bootstrap required, opened a new log
I20260925 15:04:16.726162 25098 tablet_bootstrap.cc:492] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3: No bootstrap required, opened a new log
I20260925 15:04:16.726286 25100 ts_tablet_manager.cc:1397] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260925 15:04:16.726337 25098 ts_tablet_manager.cc:1397] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.006s
I20260925 15:04:16.727114 25098 raft_consensus.cc:348] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.727339 25098 raft_consensus.cc:374] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:16.727422 25098 raft_consensus.cc:729] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35825b136bc4e28b16ac0860a8e99a3, State: Initialized, Role: FOLLOWER
I20260925 15:04:16.727164 25100 raft_consensus.cc:348] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.727675 25100 raft_consensus.cc:374] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:16.727686 25098 consensus_queue.cc:260] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.727819 25100 raft_consensus.cc:729] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3323fc84267c41a8931cc3aa93fe1bfe, State: Initialized, Role: FOLLOWER
I20260925 15:04:16.728183 25100 consensus_queue.cc:260] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.728344 25099 ts_tablet_manager.cc:1428] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260925 15:04:16.728118 25105 tablet_replica.cc:439] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } } }
I20260925 15:04:16.728452 25105 tablet_replica.cc:442] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:04:16.729355 25100 ts_tablet_manager.cc:1428] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:04:16.729432 25106 tablet_replica.cc:439] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } } }
I20260925 15:04:16.729688 25106 tablet_replica.cc:442] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:04:16.730217 25098 ts_tablet_manager.cc:1428] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260925 15:04:16.899353 25104 raft_consensus.cc:482] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:04:16.899570 25104 raft_consensus.cc:504] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.900198 25104 leader_election.cc:288] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3323fc84267c41a8931cc3aa93fe1bfe (127.22.236.1:44881), a35825b136bc4e28b16ac0860a8e99a3 (127.22.236.3:38931)
I20260925 15:04:16.900799 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d4ff44473e44eba414e62f6f833bba" candidate_uuid: "a7c5037cbbc145bdb51997721a35701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35825b136bc4e28b16ac0860a8e99a3" is_pre_election: true
I20260925 15:04:16.900872 24752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d4ff44473e44eba414e62f6f833bba" candidate_uuid: "a7c5037cbbc145bdb51997721a35701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" is_pre_election: true
I20260925 15:04:16.901052 25029 raft_consensus.cc:2454] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7c5037cbbc145bdb51997721a35701e in term 0.
I20260925 15:04:16.901081 24752 raft_consensus.cc:2454] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7c5037cbbc145bdb51997721a35701e in term 0.
I20260925 15:04:16.901484 24822 leader_election.cc:302] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3323fc84267c41a8931cc3aa93fe1bfe, a35825b136bc4e28b16ac0860a8e99a3, a7c5037cbbc145bdb51997721a35701e; no voters:
I20260925 15:04:16.901757 25104 raft_consensus.cc:2781] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:04:16.901851 25104 raft_consensus.cc:482] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:04:16.901960 25104 raft_consensus.cc:3037] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:16.903831 25104 raft_consensus.cc:504] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.904381 25104 leader_election.cc:288] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [CANDIDATE]: Term 1 election: Requested vote from peers 3323fc84267c41a8931cc3aa93fe1bfe (127.22.236.1:44881), a35825b136bc4e28b16ac0860a8e99a3 (127.22.236.3:38931)
I20260925 15:04:16.904868 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d4ff44473e44eba414e62f6f833bba" candidate_uuid: "a7c5037cbbc145bdb51997721a35701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35825b136bc4e28b16ac0860a8e99a3"
I20260925 15:04:16.904906 24752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d4ff44473e44eba414e62f6f833bba" candidate_uuid: "a7c5037cbbc145bdb51997721a35701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3323fc84267c41a8931cc3aa93fe1bfe"
I20260925 15:04:16.905071 25029 raft_consensus.cc:3037] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:16.905112 24752 raft_consensus.cc:3037] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:16.906986 25029 raft_consensus.cc:2454] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c5037cbbc145bdb51997721a35701e in term 1.
I20260925 15:04:16.906982 24752 raft_consensus.cc:2454] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c5037cbbc145bdb51997721a35701e in term 1.
I20260925 15:04:16.907366 24822 leader_election.cc:302] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3323fc84267c41a8931cc3aa93fe1bfe, a35825b136bc4e28b16ac0860a8e99a3, a7c5037cbbc145bdb51997721a35701e; no voters:
I20260925 15:04:16.907612 25104 raft_consensus.cc:2781] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:16.907787 25104 raft_consensus.cc:686] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [term 1 LEADER]: Becoming Leader. State: Replica: a7c5037cbbc145bdb51997721a35701e, State: Running, Role: LEADER
I20260925 15:04:16.908010 25104 consensus_queue.cc:237] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } }
I20260925 15:04:16.909431 25129 tablet_replica.cc:439] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e: TxnStatusTablet state changed. Reason: New leader a7c5037cbbc145bdb51997721a35701e. Latest consensus state: current_term: 1 leader_uuid: "a7c5037cbbc145bdb51997721a35701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } } }
I20260925 15:04:16.910215 25129 tablet_replica.cc:442] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:04:16.910723 25142 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:04:16.910880 25142 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:04:16.910807 24593 catalog_manager.cc:5495] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e reported cstate change: term changed from 0 to 1, leader changed from <none> to a7c5037cbbc145bdb51997721a35701e (127.22.236.2). New cstate: current_term: 1 leader_uuid: "a7c5037cbbc145bdb51997721a35701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } health_report { overall_health: UNKNOWN } } }
I20260925 15:04:16.965547 25136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:16.958700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41847 (local address 127.0.0.1:59670)
0925 15:04:16.958963 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:16.958986 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:16.959123 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:16.960129 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:16.960132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:16.960164 (+ 32us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:04:16.960323 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:16.960331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.961814 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.961818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:16.962794 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:16.962805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.963860 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.963864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:16.963988 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:16.964039 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:04:16.965139 (+ 1100us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:04:16.965151 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 15:04:16.965201 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 15:04:16.965328 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 15:04:16.965654 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:16.958660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.22.236.2:41847)
0925 15:04:16.959728 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:16.959733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:16.959753 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:04:16.959849 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:16.959854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:16.959964 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:16.960126 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:16.960457 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.961693 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.962920 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.963682 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.963854 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:16.964269 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:16.965043 (+ 774us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:16.965191 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:16.965245 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 15:04:16.965322 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":401,"threads_started":1}
I20260925 15:04:16.970551 25029 raft_consensus.cc:1261] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3 [term 1 FOLLOWER]: Refusing update from remote peer a7c5037cbbc145bdb51997721a35701e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:16.970814 24752 raft_consensus.cc:1261] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe [term 1 FOLLOWER]: Refusing update from remote peer a7c5037cbbc145bdb51997721a35701e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:16.971248 25104 consensus_queue.cc:1035] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:16.971570 25129 consensus_queue.cc:1035] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:16.974969 25106 tablet_replica.cc:439] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe: TxnStatusTablet state changed. Reason: New leader a7c5037cbbc145bdb51997721a35701e. Latest consensus state: current_term: 1 leader_uuid: "a7c5037cbbc145bdb51997721a35701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } } }
I20260925 15:04:16.975272 25106 tablet_replica.cc:442] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:04:16.978374 25104 tablet_replica.cc:439] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7c5037cbbc145bdb51997721a35701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } } }
I20260925 15:04:16.978691 25104 tablet_replica.cc:442] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:04:16.980185 25106 tablet_replica.cc:439] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7c5037cbbc145bdb51997721a35701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } } }
I20260925 15:04:16.980415 25106 tablet_replica.cc:442] T 92d4ff44473e44eba414e62f6f833bba P 3323fc84267c41a8931cc3aa93fe1bfe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:04:16.984756 25105 tablet_replica.cc:439] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3: TxnStatusTablet state changed. Reason: New leader a7c5037cbbc145bdb51997721a35701e. Latest consensus state: current_term: 1 leader_uuid: "a7c5037cbbc145bdb51997721a35701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } } }
I20260925 15:04:16.985041 25105 tablet_replica.cc:442] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:04:16.986392 25129 tablet_replica.cc:439] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7c5037cbbc145bdb51997721a35701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } } }
I20260925 15:04:16.986635 25129 tablet_replica.cc:442] T 92d4ff44473e44eba414e62f6f833bba P a7c5037cbbc145bdb51997721a35701e: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:04:16.990190 25105 tablet_replica.cc:439] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7c5037cbbc145bdb51997721a35701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 } } peers { permanent_uuid: "a7c5037cbbc145bdb51997721a35701e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41847 } } peers { permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 } } }
I20260925 15:04:16.990388 25105 tablet_replica.cc:442] T 92d4ff44473e44eba414e62f6f833bba P a35825b136bc4e28b16ac0860a8e99a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:04:16.994052 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:16.987940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.22.236.2:41847)
0925 15:04:16.988101 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:16.988108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:16.988679 (+ 571us) server_negotiation.cc:407] Connection header received
0925 15:04:16.988840 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:16.988845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:16.988919 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:16.989000 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:16.989629 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.991191 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.992538 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:16.992965 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:16.993066 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:16.993433 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:16.993534 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:16.993602 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:16.993731 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 15:04:16.993802 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:04:17.010535 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:17.004385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41847 (local address 127.22.236.2:53125)
0925 15:04:17.005089 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:17.005108 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:17.005224 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:17.005557 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:17.005560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:17.005569 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:04:17.005874 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:17.005881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:17.007263 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:17.007266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:17.008497 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:17.008509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:17.009500 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:17.009504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:17.009612 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:17.009664 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:04:17.010056 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:04:17.010063 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:04:17.010116 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:04:17.010205 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":414,"threads_started":1}
I20260925 15:04:17.010562 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:17.004375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:53125 (local address 127.22.236.2:41847)
0925 15:04:17.004560 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:17.004566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:17.005198 (+ 632us) server_negotiation.cc:407] Connection header received
0925 15:04:17.005337 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:17.005343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:17.005447 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:17.005532 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:17.006002 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:17.007137 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:17.008621 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:17.009338 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:17.009465 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:17.009840 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:17.009965 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:17.010050 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:17.010155 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 15:04:17.010227 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:04:17.019551 25029 raft_consensus.cc:1261] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3 [term 1 FOLLOWER]: Refusing update from remote peer a7c5037cbbc145bdb51997721a35701e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:17.019567 24752 raft_consensus.cc:1261] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe [term 1 FOLLOWER]: Refusing update from remote peer a7c5037cbbc145bdb51997721a35701e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:17.020061 25104 consensus_queue.cc:1035] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3323fc84267c41a8931cc3aa93fe1bfe" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:17.020361 25159 consensus_queue.cc:1035] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a35825b136bc4e28b16ac0860a8e99a3" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:17.026968 25147 mvcc.cc:205] Tried to move back new op lower bound from 7333268099147259904 to 7333268097527779328. Current Snapshot: MvccSnapshot[applied={T|T < 7333268099147259904}]
I20260925 15:04:17.082510 24593 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:17.082723 24593 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:17.087867 24593 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8c0bd9e4c434f4bb6f0e649b07dd654: Sending DeleteTablet for 3 replicas of tablet cfead76f9b9047e2a9517a90408b853f
I20260925 15:04:17.088749 24730 tablet_service.cc:1514] Processing DeleteTablet for tablet cfead76f9b9047e2a9517a90408b853f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:17 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:53012
I20260925 15:04:17.093451 25009 tablet_service.cc:1514] Processing DeleteTablet for tablet cfead76f9b9047e2a9517a90408b853f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:17 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:57226
I20260925 15:04:17.094264 24870 tablet_service.cc:1514] Processing DeleteTablet for tablet cfead76f9b9047e2a9517a90408b853f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:17 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:33390
I20260925 15:04:17.101205 25163 tablet_replica.cc:330] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe: stopping tablet replica
I20260925 15:04:17.101415 25163 raft_consensus.cc:2229] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:17.101670 25163 raft_consensus.cc:2258] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:17.102723 25163 ts_tablet_manager.cc:1905] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:17.109129 25163 ts_tablet_manager.cc:1918] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:04:17.109249 25163 log.cc:1196] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/cfead76f9b9047e2a9517a90408b853f
I20260925 15:04:17.109632 25163 ts_tablet_manager.cc:1939] T cfead76f9b9047e2a9517a90408b853f P 3323fc84267c41a8931cc3aa93fe1bfe: Deleting consensus metadata
I20260925 15:04:17.110646 24579 catalog_manager.cc:4841] TS 3323fc84267c41a8931cc3aa93fe1bfe (127.22.236.1:44881): tablet cfead76f9b9047e2a9517a90408b853f (table test-table [id=142b1676259b46ce9b723af9c23f7ee1]) successfully deleted
I20260925 15:04:17.111646 25166 tablet_replica.cc:330] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e: stopping tablet replica
I20260925 15:04:17.111871 25166 raft_consensus.cc:2229] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:04:17.112499 25166 raft_consensus.cc:2258] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:17.112983 25165 tablet_replica.cc:330] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3: stopping tablet replica
I20260925 15:04:17.113166 25165 raft_consensus.cc:2229] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:17.113375 25165 raft_consensus.cc:2258] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:17.113710 25166 ts_tablet_manager.cc:1905] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:17.114414 25165 ts_tablet_manager.cc:1905] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:17.118729 25166 ts_tablet_manager.cc:1918] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:04:17.118840 25166 log.cc:1196] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/cfead76f9b9047e2a9517a90408b853f
I20260925 15:04:17.119181 25166 ts_tablet_manager.cc:1939] T cfead76f9b9047e2a9517a90408b853f P a7c5037cbbc145bdb51997721a35701e: Deleting consensus metadata
I20260925 15:04:17.120047 24579 catalog_manager.cc:4841] TS a7c5037cbbc145bdb51997721a35701e (127.22.236.2:41847): tablet cfead76f9b9047e2a9517a90408b853f (table test-table [id=142b1676259b46ce9b723af9c23f7ee1]) successfully deleted
I20260925 15:04:17.121498 25165 ts_tablet_manager.cc:1918] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:04:17.121610 25165 log.cc:1196] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/cfead76f9b9047e2a9517a90408b853f
I20260925 15:04:17.121989 25165 ts_tablet_manager.cc:1939] T cfead76f9b9047e2a9517a90408b853f P a35825b136bc4e28b16ac0860a8e99a3: Deleting consensus metadata
I20260925 15:04:17.122879 24580 catalog_manager.cc:4841] TS a35825b136bc4e28b16ac0860a8e99a3 (127.22.236.3:38931): tablet cfead76f9b9047e2a9517a90408b853f (table test-table [id=142b1676259b46ce9b723af9c23f7ee1]) successfully deleted
Sep 25 15:04:17 dist-test-slave-nsrc krb5kdc[24550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:04:17.131711 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:17.104623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.22.236.1:44881)
0925 15:04:17.104836 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:17.104842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:17.105549 (+ 707us) server_negotiation.cc:407] Connection header received
0925 15:04:17.105810 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:17.105816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:17.105898 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:17.105968 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:17.108290 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:17.109355 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:17.111293 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:17.111889 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:17.112014 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:17.125362 (+ 13348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:17.125375 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:17.125388 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:17.125424 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:17.128914 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:17.129532 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:17.129541 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:17.129545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:17.129692 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:17.130033 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:17.130037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:17.130041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:17.130484 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:17.130766 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:17.131253 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 15:04:17.131444 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 15:04:17.132395 24796 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41544
I20260925 15:04:17.154206 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:17.141236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.22.236.62:33711)
0925 15:04:17.141459 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:17.141465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:17.141719 (+ 254us) server_negotiation.cc:407] Connection header received
0925 15:04:17.141884 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:17.141888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:17.141966 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:17.142043 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:17.143608 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:17.144375 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:17.146094 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:17.146506 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:17.146591 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:17.149204 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:17.149208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:17.149211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:17.149240 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:17.151844 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:17.152484 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:17.152490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:17.152494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:17.152606 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:17.152946 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:17.152950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:17.152954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:17.153208 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:17.153430 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:17.153713 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 15:04:17.153914 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260925 15:04:17.154737 24593 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58374
I20260925 15:04:17.156675 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 24666
I20260925 15:04:17.169571 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:17.157862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.22.236.1:44881)
0925 15:04:17.158030 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:17.158036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:17.158054 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:04:17.158133 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:17.158137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:17.158200 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:17.158293 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:17.159251 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:17.160365 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:17.161456 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:17.162029 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:17.162183 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:17.164093 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:17.164100 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:17.164105 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:17.164173 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:17.167226 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:17.167778 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:17.167783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:17.167786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:17.167928 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:17.168226 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:17.168230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:17.168234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:17.168558 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:17.168782 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:17.169010 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 15:04:17.169222 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 15:04:17.170084 24796 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41554
W20260925 15:04:17.170565 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 24666: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:04:17.170732 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 24666
I20260925 15:04:17.187325 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 24805
Sep 25 15:04:17 dist-test-slave-nsrc krb5kdc[24550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:04:17.201638 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:17.188025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.22.236.2:41847)
0925 15:04:17.188232 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:17.188238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:17.188256 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:04:17.188438 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:17.188444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:17.188511 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:17.188599 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:17.189676 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:17.190401 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:17.191594 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:17.192098 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:17.192333 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:17.196254 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:17.196265 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:17.196277 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:17.196314 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:17.199065 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:17.199660 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:17.199666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:17.199669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:17.199794 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:17.200166 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:17.200172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:17.200175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:17.200557 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:17.200859 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:17.201124 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 15:04:17.201342 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 15:04:17.202266 24935 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59688
W20260925 15:04:17.202888 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 24805: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:04:17.203037 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 24805
I20260925 15:04:17.217823 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 24944
Sep 25 15:04:17 dist-test-slave-nsrc krb5kdc[24550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:04:17.231294 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:17.218434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.22.236.3:38931)
0925 15:04:17.218937 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:17.218942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:17.218957 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:04:17.219006 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:17.219010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:17.219063 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:17.219148 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:17.220304 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:17.221076 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:17.222240 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:17.222731 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:17.222848 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:17.226615 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:17.226626 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:17.226642 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:17.226674 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:17.229086 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:17.229621 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:17.229627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:17.229630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:17.229752 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:17.230016 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:17.230019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:17.230021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:17.230322 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:17.230523 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:17.230808 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 15:04:17.230984 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":285,"threads_started":1}
W20260925 15:04:17.231909 25074 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47892
W20260925 15:04:17.232559 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 24944: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:04:17.232676 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 24944
I20260925 15:04:17.245599 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 24563
I20260925 15:04:17.246532 24635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:17.355248 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 24563
2026-09-25T15:04:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4625 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:04:17 dist-test-slave-nsrc krb5kdc[25175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:04:17 dist-test-slave-nsrc krb5kdc[25175](info): set up 2 sockets
Sep 25 15:04:17 dist-test-slave-nsrc krb5kdc[25175](info): commencing operation
krb5kdc: starting...
W20260925 15:04:18.343983 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:04:18 dist-test-slave-nsrc krb5kdc[25175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:04:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:04:18.510146 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:38627
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:38627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:18.810748 25188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:18.811017 25188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:18.811136 25188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:18.818244 25188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:18.818343 25188 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:04:18.818405 25188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:18.818490 25188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:18.818544 25188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:18.818624 25188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:04:18.828207 25188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:38627
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:38627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:18.829399 25188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:18.830646 25188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:18.841832 25193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:18.841888 25196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:18.841922 25194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:18.842680 25188 server_base.cc:1034] running on GCE node
I20260925 15:04:18.843355 25188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:18.844713 25188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:18.845927 25188 hybrid_clock.cc:648] HybridClock initialized: now 1790348658845875 us; error 29 us; skew 500 ppm
I20260925 15:04:18.846215 25188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:18 dist-test-slave-nsrc krb5kdc[25175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348658, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:18.849048 25188 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:04:18.850378 25188 webserver.cc:460] Webserver started at http://127.22.236.62:36045/ using document root <none> and password file <none>
I20260925 15:04:18.851016 25188 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:18.851109 25188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:18.851392 25188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:18.854202 25188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "e26e7b457a724a06aefec4b763906182"
format_stamp: "Formatted at 2026-09-25 15:04:18 on dist-test-slave-nsrc"
server_key: "2c922da74664e0d5c578a7a55a662fe2"
server_key_iv: "553d32251ce904834eb4e36664199715"
server_key_version: "encryptionkey@0"
I20260925 15:04:18.854913 25188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "e26e7b457a724a06aefec4b763906182"
format_stamp: "Formatted at 2026-09-25 15:04:18 on dist-test-slave-nsrc"
server_key: "2c922da74664e0d5c578a7a55a662fe2"
server_key_iv: "553d32251ce904834eb4e36664199715"
server_key_version: "encryptionkey@0"
I20260925 15:04:18.859529 25188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 15:04:18.862684 25203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:18.863886 25188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:04:18.864029 25188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "e26e7b457a724a06aefec4b763906182"
format_stamp: "Formatted at 2026-09-25 15:04:18 on dist-test-slave-nsrc"
server_key: "2c922da74664e0d5c578a7a55a662fe2"
server_key_iv: "553d32251ce904834eb4e36664199715"
server_key_version: "encryptionkey@0"
I20260925 15:04:18.864238 25188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:18.889014 25188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:18.892567 25188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:18.892791 25188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:18.911352 25188 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:38627
I20260925 15:04:18.911374 25264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:38627 every 8 connection(s)
I20260925 15:04:18.912572 25188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:18.913738 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 25188
I20260925 15:04:18.913947 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:18.914718 23472 external_mini_cluster.cc:1428] Setting key 06b8078d6c4ecaffef528d8f704c05c8
I20260925 15:04:18.916908 25265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:04:18 dist-test-slave-nsrc krb5kdc[25175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:18.930810 25265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182: Bootstrap starting.
I20260925 15:04:18.933868 25265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:18.935081 25265 log.cc:824] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:18.938057 25265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182: No bootstrap required, opened a new log
I20260925 15:04:18.938203 25268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:18.916672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.22.236.62:38627)
0925 15:04:18.917330 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:18.917345 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:18.917393 (+ 48us) server_negotiation.cc:407] Connection header received
0925 15:04:18.919437 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:18.919463 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:18.919955 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:18.920394 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:18.921573 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:18.923526 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:18.925351 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:18.926338 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:18.926615 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:18.931302 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:18.931310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:18.931324 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:18.931361 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:18.934214 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:18.934969 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:18.934975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:18.934981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:18.935137 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:18.935516 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:18.935520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:18.935523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:18.936244 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:18.936480 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:18.936885 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 15:04:18.937196 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":273,"threads_started":1}
I20260925 15:04:18.945876 25265 raft_consensus.cc:348] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26e7b457a724a06aefec4b763906182" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38627 } }
I20260925 15:04:18.946179 25265 raft_consensus.cc:374] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:18.946270 25265 raft_consensus.cc:729] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26e7b457a724a06aefec4b763906182, State: Initialized, Role: FOLLOWER
I20260925 15:04:18.946811 25265 consensus_queue.cc:260] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26e7b457a724a06aefec4b763906182" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38627 } }
I20260925 15:04:18.947006 25265 raft_consensus.cc:388] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:18.947129 25265 raft_consensus.cc:482] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:18.947270 25265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:18.949280 25265 raft_consensus.cc:504] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26e7b457a724a06aefec4b763906182" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38627 } }
I20260925 15:04:18.949720 25265 leader_election.cc:302] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e26e7b457a724a06aefec4b763906182; no voters:
I20260925 15:04:18.950217 25265 leader_election.cc:288] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:18.950335 25270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:18.950917 25270 raft_consensus.cc:686] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [term 1 LEADER]: Becoming Leader. State: Replica: e26e7b457a724a06aefec4b763906182, State: Running, Role: LEADER
I20260925 15:04:18.951424 25270 consensus_queue.cc:237] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26e7b457a724a06aefec4b763906182" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38627 } }
I20260925 15:04:18.952041 25265 sys_catalog.cc:564] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:18.954139 25272 sys_catalog.cc:455] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e26e7b457a724a06aefec4b763906182. Latest consensus state: current_term: 1 leader_uuid: "e26e7b457a724a06aefec4b763906182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26e7b457a724a06aefec4b763906182" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38627 } } }
I20260925 15:04:18.954504 25272 sys_catalog.cc:458] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:18.955240 25271 sys_catalog.cc:455] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e26e7b457a724a06aefec4b763906182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26e7b457a724a06aefec4b763906182" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38627 } } }
I20260925 15:04:18.955505 25271 sys_catalog.cc:458] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:18.957039 25278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:18.963258 25278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:18.971891 25278 catalog_manager.cc:1347] Generated new cluster ID: dbbce363462d41e8a08172403cbaaba8
I20260925 15:04:18.971994 25278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:18.989652 25278 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:18.990934 25278 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:18.997944 25278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182: Generated new TSK 0
I20260925 15:04:18.998692 25278 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:04:19.043241 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:38627
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:19.370347 25291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:19.370609 25291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:19.370707 25291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:19.377648 25291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:19.377759 25291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:19.377847 25291 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:04:19.377897 25291 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:04:19.377964 25291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:04:19.387434 25291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.236.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:19.388643 25291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:19.389858 25291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:19.401852 25299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:19.401829 25297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:19.401909 25296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:19.402563 25291 server_base.cc:1034] running on GCE node
I20260925 15:04:19.403131 25291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:19.403986 25291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:19.405182 25291 hybrid_clock.cc:648] HybridClock initialized: now 1790348659405125 us; error 43 us; skew 500 ppm
I20260925 15:04:19.405490 25291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:19 dist-test-slave-nsrc krb5kdc[25175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348659, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:19.408506 25291 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:04:19.409904 25291 webserver.cc:460] Webserver started at http://127.22.236.1:40387/ using document root <none> and password file <none>
I20260925 15:04:19.410550 25291 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:19.410662 25291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:19.410946 25291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:19.413738 25291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "1f9d90c6ae3548c5b66bef265e5b34ab"
format_stamp: "Formatted at 2026-09-25 15:04:19 on dist-test-slave-nsrc"
server_key: "f38c0f4e9255a5b94d95bb6808663ccc"
server_key_iv: "74a52f0812fdc37c19ceae76b258664c"
server_key_version: "encryptionkey@0"
I20260925 15:04:19.414433 25291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "1f9d90c6ae3548c5b66bef265e5b34ab"
format_stamp: "Formatted at 2026-09-25 15:04:19 on dist-test-slave-nsrc"
server_key: "f38c0f4e9255a5b94d95bb6808663ccc"
server_key_iv: "74a52f0812fdc37c19ceae76b258664c"
server_key_version: "encryptionkey@0"
I20260925 15:04:19.419106 25291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:04:19.422276 25306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:19.423480 25291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:04:19.423616 25291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "1f9d90c6ae3548c5b66bef265e5b34ab"
format_stamp: "Formatted at 2026-09-25 15:04:19 on dist-test-slave-nsrc"
server_key: "f38c0f4e9255a5b94d95bb6808663ccc"
server_key_iv: "74a52f0812fdc37c19ceae76b258664c"
server_key_version: "encryptionkey@0"
I20260925 15:04:19.423785 25291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:19.447804 25291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:19.451946 25291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:19.452246 25291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:19.454811 25291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:19.454926 25291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:19.455034 25291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:19.455112 25291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:04:19 dist-test-slave-nsrc krb5kdc[25175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348659, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:19.476456 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:19.455367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:39055 (local address 127.22.236.62:38627)
0925 15:04:19.456308 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:19.456320 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:19.457099 (+ 779us) server_negotiation.cc:407] Connection header received
0925 15:04:19.459170 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:19.459176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:19.459263 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:19.459384 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:19.461750 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.462610 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.465249 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.465673 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.465768 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:19.470579 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:19.470584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:19.470588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:19.470618 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:19.473263 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:19.474129 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:19.474132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:19.474135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:19.474245 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:19.474687 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:19.474690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:19.474692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:19.474953 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:19.475057 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:19.475912 (+ 855us) server_negotiation.cc:299] Negotiation successful
0925 15:04:19.476068 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":778,"threads_started":1}
I20260925 15:04:19.478202 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:19.456076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38627 (local address 127.22.236.1:39055)
0925 15:04:19.456965 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:19.457004 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:19.458882 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:19.459681 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:19.459697 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:19.460567 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:19.461536 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:19.461572 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.462772 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.462778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:19.465096 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:19.465112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.465905 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.465912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:19.466197 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:19.466906 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:19.466937 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:19.470298 (+ 3361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:19.473391 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:19.473401 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:19.473422 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:19.474014 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:19.474336 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:19.474341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:19.474345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:19.474591 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:19.475063 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:19.475072 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:19.475546 (+ 474us) client_negotiation.cc:769] Sending connection context
0925 15:04:19.475849 (+ 303us) client_negotiation.cc:240] Negotiation successful
0925 15:04:19.476470 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":383,"threads_started":1}
I20260925 15:04:19.497684 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:19.484403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:39629 (local address 127.22.236.62:38627)
0925 15:04:19.484567 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:19.484573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:19.485376 (+ 803us) server_negotiation.cc:407] Connection header received
0925 15:04:19.485560 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:19.485566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:19.485631 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:19.485717 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:19.486594 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.487280 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.488663 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.489081 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.489190 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:19.491453 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:19.491460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:19.491464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:19.491498 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:19.494835 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:19.495468 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:19.495474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:19.495478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:19.495641 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:19.496070 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:19.496074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:19.496076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:19.496440 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:19.496625 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:19.497106 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 15:04:19.497344 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:04:19.497767 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:19.484539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38627 (local address 127.22.236.1:39629)
0925 15:04:19.485267 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:19.485285 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:19.485448 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:19.485755 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:19.485759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:19.486188 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:19.486463 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:19.486472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.487416 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.487421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:19.488540 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:19.488552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.489259 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.489266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:19.489459 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:19.490210 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:19.490233 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:19.491288 (+ 1055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:19.494989 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:19.494993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:19.494996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:19.495364 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:19.495732 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:19.495737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:19.495740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:19.495924 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:19.496585 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:19.496590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:19.496805 (+ 215us) client_negotiation.cc:769] Sending connection context
0925 15:04:19.496990 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 15:04:19.497297 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":415,"threads_started":1}
I20260925 15:04:19.507426 25291 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:34715
I20260925 15:04:19.507452 25426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:34715 every 8 connection(s)
I20260925 15:04:19.508643 25291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:04:19.510281 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 25291
I20260925 15:04:19.510461 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:04:19.511070 23472 external_mini_cluster.cc:1428] Setting key d9a62564b87f8f9367bf9142224c16e6
I20260925 15:04:19.510504 25427 heartbeater.cc:347] Connected to a master server at 127.22.236.62:38627
I20260925 15:04:19.511840 25427 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:19.512511 25427 heartbeater.cc:511] Master 127.22.236.62:38627 requested a full tablet report, sending...
I20260925 15:04:19.514503 25220 ts_manager.cc:194] Registered new tserver with Master: 1f9d90c6ae3548c5b66bef265e5b34ab (127.22.236.1:34715)
I20260925 15:04:19.516343 25220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:39055
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:04:19.547900 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:38627
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:19.846722 25431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:19.846977 25431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:19.847106 25431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:19.854411 25431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:19.854512 25431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:19.854630 25431 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:04:19.854688 25431 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:04:19.854792 25431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:04:19.864434 25431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.236.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:19.865620 25431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:19.866847 25431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:19.878660 25436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:19.878707 25439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:19.878732 25437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:19.879431 25431 server_base.cc:1034] running on GCE node
I20260925 15:04:19.880017 25431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:19.880913 25431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:19.882103 25431 hybrid_clock.cc:648] HybridClock initialized: now 1790348659882053 us; error 43 us; skew 500 ppm
I20260925 15:04:19.882377 25431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:19 dist-test-slave-nsrc krb5kdc[25175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348659, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:19.885432 25431 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:04:19.886693 25431 webserver.cc:460] Webserver started at http://127.22.236.2:33255/ using document root <none> and password file <none>
I20260925 15:04:19.887288 25431 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:19.887403 25431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:19.887674 25431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:19.890499 25431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "4dc66b1554ef4cb2b5195429e74b8e19"
format_stamp: "Formatted at 2026-09-25 15:04:19 on dist-test-slave-nsrc"
server_key: "b9a02900f6990db0f704e8e8ca105da1"
server_key_iv: "67533eaeb14073a4a7295677d719594c"
server_key_version: "encryptionkey@0"
I20260925 15:04:19.891201 25431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "4dc66b1554ef4cb2b5195429e74b8e19"
format_stamp: "Formatted at 2026-09-25 15:04:19 on dist-test-slave-nsrc"
server_key: "b9a02900f6990db0f704e8e8ca105da1"
server_key_iv: "67533eaeb14073a4a7295677d719594c"
server_key_version: "encryptionkey@0"
I20260925 15:04:19.895875 25431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260925 15:04:19.899057 25446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:19.900347 25431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:19.900480 25431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "4dc66b1554ef4cb2b5195429e74b8e19"
format_stamp: "Formatted at 2026-09-25 15:04:19 on dist-test-slave-nsrc"
server_key: "b9a02900f6990db0f704e8e8ca105da1"
server_key_iv: "67533eaeb14073a4a7295677d719594c"
server_key_version: "encryptionkey@0"
I20260925 15:04:19.900640 25431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:19.926913 25431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:19.930456 25431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:19.930697 25431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:19.933131 25431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:19.933228 25431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:19.933350 25431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:19.933425 25431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:04:19 dist-test-slave-nsrc krb5kdc[25175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348659, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:19.954588 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:19.933454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:57673 (local address 127.22.236.62:38627)
0925 15:04:19.933654 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:19.933661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:19.935045 (+ 1384us) server_negotiation.cc:407] Connection header received
0925 15:04:19.937176 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:19.937182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:19.937267 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:19.937367 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:19.939618 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.940600 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.942720 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.943296 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.943425 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:19.947667 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:19.947674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:19.947678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:19.947717 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:19.950998 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:19.951910 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:19.951916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:19.951919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:19.952053 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:19.952661 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:19.952665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:19.952668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:19.953038 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:19.953198 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:19.954048 (+ 850us) server_negotiation.cc:299] Negotiation successful
0925 15:04:19.954249 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:04:19.955538 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:19.934063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38627 (local address 127.22.236.2:57673)
0925 15:04:19.934868 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:19.934924 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:19.936884 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:19.937652 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:19.937664 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:19.938488 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:19.939411 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:19.939443 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.940744 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.940750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:19.942525 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:19.942540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.943567 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.943578 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:19.943873 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:19.944546 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:19.944583 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:19.947402 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:19.951160 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:19.951169 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:19.951190 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:19.951784 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:19.952177 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:19.952183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:19.952187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:19.952496 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:19.953193 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:19.953204 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:19.953556 (+ 352us) client_negotiation.cc:769] Sending connection context
0925 15:04:19.953945 (+ 389us) client_negotiation.cc:240] Negotiation successful
0925 15:04:19.954438 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":337,"threads_started":1}
I20260925 15:04:19.974040 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:19.960506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38627 (local address 127.22.236.2:33569)
0925 15:04:19.961266 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:19.961283 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:19.961397 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:19.961692 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:19.961695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:19.962082 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:19.962343 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:19.962351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.963764 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.963767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:19.965078 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:19.965096 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.966113 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.966120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:19.966289 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:19.966956 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:19.966976 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:19.968006 (+ 1030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:19.971600 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:19.971604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:19.971608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:19.972126 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:19.972536 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:19.972539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:19.972541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:19.972651 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:19.973259 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:19.973262 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:19.973404 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 15:04:19.973515 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:04:19.973722 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":483,"threads_started":1}
I20260925 15:04:19.974185 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:19.960448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:33569 (local address 127.22.236.62:38627)
0925 15:04:19.960616 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:19.960622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:19.961362 (+ 740us) server_negotiation.cc:407] Connection header received
0925 15:04:19.961493 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:19.961499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:19.961574 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:19.961676 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:19.962472 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.963643 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.965254 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:19.965770 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:19.965840 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:19.968225 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:19.968231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:19.968236 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:19.968269 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:19.971455 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:19.972257 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:19.972263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:19.972268 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:19.972437 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:19.972736 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:19.972741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:19.972745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:19.973159 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:19.973303 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:19.973631 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 15:04:19.973860 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:04:19.979514 25431 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:41355
I20260925 15:04:19.979544 25565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:41355 every 8 connection(s)
I20260925 15:04:19.980747 25431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:04:19.982594 25566 heartbeater.cc:347] Connected to a master server at 127.22.236.62:38627
I20260925 15:04:19.982621 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 25431
I20260925 15:04:19.982769 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:04:19.982810 25566 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:19.983295 23472 external_mini_cluster.cc:1428] Setting key 938a032adcb3279add2ec2c2e03a778b
I20260925 15:04:19.983503 25566 heartbeater.cc:511] Master 127.22.236.62:38627 requested a full tablet report, sending...
I20260925 15:04:19.984882 25219 ts_manager.cc:194] Registered new tserver with Master: 4dc66b1554ef4cb2b5195429e74b8e19 (127.22.236.2:41355)
I20260925 15:04:19.985679 25219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:57673
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:04:20.017338 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:38627
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:20.317799 25570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:20.318060 25570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:20.318166 25570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:20.325181 25570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:20.325289 25570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:20.325377 25570 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:04:20.325425 25570 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:04:20.325491 25570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:04:20.334934 25570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.236.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:20.336072 25570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:20.337357 25570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:20.349458 25576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:20.349534 25578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:20.350078 25575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:20.350445 25570 server_base.cc:1034] running on GCE node
I20260925 15:04:20.350991 25570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:20.351891 25570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:20.353086 25570 hybrid_clock.cc:648] HybridClock initialized: now 1790348660353035 us; error 43 us; skew 500 ppm
I20260925 15:04:20.353386 25570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:20 dist-test-slave-nsrc krb5kdc[25175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348660, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:20.356238 25570 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:04:20.357641 25570 webserver.cc:460] Webserver started at http://127.22.236.3:44031/ using document root <none> and password file <none>
I20260925 15:04:20.358276 25570 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:20.358397 25570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:20.358671 25570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:20.361487 25570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "0ab5582485d84191bde61fd1c9a18585"
format_stamp: "Formatted at 2026-09-25 15:04:20 on dist-test-slave-nsrc"
server_key: "7898bf712a0844d4a53862cd76b053c1"
server_key_iv: "ccfea2390539062c2ff7fae1d2a23420"
server_key_version: "encryptionkey@0"
I20260925 15:04:20.362179 25570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "0ab5582485d84191bde61fd1c9a18585"
format_stamp: "Formatted at 2026-09-25 15:04:20 on dist-test-slave-nsrc"
server_key: "7898bf712a0844d4a53862cd76b053c1"
server_key_iv: "ccfea2390539062c2ff7fae1d2a23420"
server_key_version: "encryptionkey@0"
I20260925 15:04:20.366956 25570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:04:20.370182 25585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:20.371459 25570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:04:20.371598 25570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "0ab5582485d84191bde61fd1c9a18585"
format_stamp: "Formatted at 2026-09-25 15:04:20 on dist-test-slave-nsrc"
server_key: "7898bf712a0844d4a53862cd76b053c1"
server_key_iv: "ccfea2390539062c2ff7fae1d2a23420"
server_key_version: "encryptionkey@0"
I20260925 15:04:20.371778 25570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:20.391010 25570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:20.394856 25570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:20.395112 25570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:20.397639 25570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:20.397758 25570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:20.397869 25570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:20.397953 25570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:04:20 dist-test-slave-nsrc krb5kdc[25175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348660, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:20.419950 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.398211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:35043 (local address 127.22.236.62:38627)
0925 15:04:20.398412 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:20.398418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:20.399849 (+ 1431us) server_negotiation.cc:407] Connection header received
0925 15:04:20.401991 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:20.401996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:20.402076 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:20.402159 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:20.404252 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.405438 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.407825 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.408482 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.408621 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.413078 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:20.413085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:20.413090 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:20.413131 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:20.416187 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:20.417569 (+ 1382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:20.417573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:20.417576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:20.417694 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:20.418246 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:20.418251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:20.418255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:20.418533 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:20.418658 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:20.419372 (+ 714us) server_negotiation.cc:299] Negotiation successful
0925 15:04:20.419601 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:04:20.421224 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.398912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38627 (local address 127.22.236.3:35043)
0925 15:04:20.399687 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:20.399730 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:20.401705 (+ 1975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:20.402449 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:20.402461 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:20.403216 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:20.404035 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.404068 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.405613 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.405622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.407660 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.407673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.408742 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.408751 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.408956 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.409763 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:20.409787 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:20.412762 (+ 2975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:20.416790 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:20.416802 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:20.416829 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:20.417404 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:20.417828 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:20.417833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:20.417836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:20.418121 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:20.418657 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:20.418665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:20.419019 (+ 354us) client_negotiation.cc:769] Sending connection context
0925 15:04:20.419247 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 15:04:20.419778 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":348,"threads_started":1}
I20260925 15:04:20.440927 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.428031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:52231 (local address 127.22.236.62:38627)
0925 15:04:20.428219 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:20.428224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:20.428757 (+ 533us) server_negotiation.cc:407] Connection header received
0925 15:04:20.428905 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:20.428911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:20.428987 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:20.429064 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:20.429918 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.430623 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.432389 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.432977 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.433119 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.434775 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:20.434781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:20.434786 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:20.434827 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:20.437955 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:20.438878 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:20.438883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:20.438887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:20.439049 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:20.439499 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:20.439504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:20.439508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:20.439789 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:20.439919 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:20.440477 (+ 558us) server_negotiation.cc:299] Negotiation successful
0925 15:04:20.440684 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:04:20.441262 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.428114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38627 (local address 127.22.236.3:52231)
0925 15:04:20.428656 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:20.428671 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:20.428795 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:20.429101 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:20.429105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:20.429525 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:20.429785 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.429794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.430749 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.430755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.432207 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.432225 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.433152 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.433156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.433311 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.433920 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:20.433936 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:20.434592 (+ 656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:20.438079 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:20.438085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:20.438090 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:20.438693 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:20.439167 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:20.439173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:20.439177 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:20.439338 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:20.439928 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:20.439933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:20.440174 (+ 241us) client_negotiation.cc:769] Sending connection context
0925 15:04:20.440340 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 15:04:20.440740 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":276,"threads_started":1}
I20260925 15:04:20.447588 25570 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:41033
I20260925 15:04:20.447615 25704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:41033 every 8 connection(s)
I20260925 15:04:20.448848 25570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:04:20.450637 25705 heartbeater.cc:347] Connected to a master server at 127.22.236.62:38627
I20260925 15:04:20.450836 25705 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:20.451436 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 25570
I20260925 15:04:20.451462 25705 heartbeater.cc:511] Master 127.22.236.62:38627 requested a full tablet report, sending...
I20260925 15:04:20.451573 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:04:20.452005 23472 external_mini_cluster.cc:1428] Setting key 52b2955b00226efe8f1248e75c9a79eb
I20260925 15:04:20.452787 25219 ts_manager.cc:194] Registered new tserver with Master: 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033)
I20260925 15:04:20.453763 25219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:35043
I20260925 15:04:20.454777 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:04:20 dist-test-slave-nsrc krb5kdc[25175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:04:20 dist-test-slave-nsrc krb5kdc[25175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:20.496083 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.483255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.22.236.62:38627)
0925 15:04:20.483406 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:20.483411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:20.483704 (+ 293us) server_negotiation.cc:407] Connection header received
0925 15:04:20.483849 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:20.483852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:20.483924 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:20.484014 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:20.485290 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.486025 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.487160 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.487554 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.487637 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.491196 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:20.491201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:20.491204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:20.491238 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:20.493747 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:20.494409 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:20.494414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:20.494417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:20.494540 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:20.494846 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:20.494849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:20.494851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:20.495137 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:20.495307 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:20.495642 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 15:04:20.495805 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:04:20.500243 25220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:04:20.507048 25220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:04:20.519491 25427 heartbeater.cc:503] Master 127.22.236.62:38627 was elected leader, sending a full tablet report...
I20260925 15:04:20.534148 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.519612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.22.236.2:41355)
0925 15:04:20.524976 (+ 5364us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:20.524987 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:20.525536 (+ 549us) server_negotiation.cc:407] Connection header received
0925 15:04:20.525781 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:20.525788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:20.525969 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:20.526114 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:20.527007 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.528429 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.531068 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.532436 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.532540 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.532748 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:20.533516 (+ 768us) server_negotiation.cc:299] Negotiation successful
0925 15:04:20.533700 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2818,"thread_start_us":2614,"threads_started":1}
I20260925 15:04:20.535243 25719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.519196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41355 (local address 127.0.0.1:50722)
0925 15:04:20.525430 (+ 6234us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:20.525448 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:20.525628 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:20.526379 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:20.526384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:20.526409 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:20.526850 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.526861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.528602 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.528606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.530890 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.530904 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.533023 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.533029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.533240 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.533294 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 15:04:20.534650 (+ 1356us) client_negotiation.cc:240] Negotiation successful
0925 15:04:20.534848 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5032,"mutex_wait_us":579,"thread_start_us":3029,"threads_started":1}
I20260925 15:04:20.536073 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.519196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:41033 (local address 127.0.0.1:40006)
0925 15:04:20.524495 (+ 5299us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:20.524547 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:20.524758 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:20.526134 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:20.526141 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:20.526174 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:20.526600 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.526613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.530005 (+ 3392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.530010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.532131 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.532168 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.535305 (+ 3137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.535313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.535501 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.535533 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:04:20.535605 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:04:20.535735 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3825,"thread_start_us":4729,"threads_started":1}
I20260925 15:04:20.537201 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.519263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40006 (local address 127.22.236.3:41033)
0925 15:04:20.525491 (+ 6228us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:20.525499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:20.525530 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:04:20.525631 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:20.525639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:20.525906 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:20.526869 (+ 963us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:20.528295 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.529848 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.533602 (+ 3754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.535066 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.536442 (+ 1376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.536673 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:20.536735 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 15:04:20.536903 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6084,"thread_start_us":5927,"threads_started":1}
I20260925 15:04:20.540215 25496 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2981cc9a2c4fa783e208e20bb830e0 (DEFAULT_TABLE table=test-table [id=4604f5859fbf4c078adb12049e0b3c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:20.541738 25496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2981cc9a2c4fa783e208e20bb830e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:20.541610 25635 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2981cc9a2c4fa783e208e20bb830e0 (DEFAULT_TABLE table=test-table [id=4604f5859fbf4c078adb12049e0b3c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:20.542440 25719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.524610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:34715 (local address 127.0.0.1:38414)
0925 15:04:20.536172 (+ 11562us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:20.536191 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:20.536307 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:20.537191 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:20.537196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:20.537221 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:20.537436 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.537446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.539232 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.539238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.540470 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.540481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.541735 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.541741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.541916 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.541947 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:04:20.542028 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:04:20.542162 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11445,"thread_start_us":13407,"threads_started":1}
I20260925 15:04:20.542491 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.524649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.22.236.1:34715)
0925 15:04:20.535894 (+ 11245us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:20.535903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:20.536418 (+ 515us) server_negotiation.cc:407] Connection header received
0925 15:04:20.536519 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:20.536526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:20.536802 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:20.536985 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:20.537589 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.539107 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.540612 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.541575 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.541701 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.541897 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:20.542083 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 15:04:20.542231 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11083,"thread_start_us":10967,"threads_started":1}
I20260925 15:04:20.542979 25635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2981cc9a2c4fa783e208e20bb830e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:20.545544 25357 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2981cc9a2c4fa783e208e20bb830e0 (DEFAULT_TABLE table=test-table [id=4604f5859fbf4c078adb12049e0b3c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:20.546576 25357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2981cc9a2c4fa783e208e20bb830e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:20.556592 25724 tablet_bootstrap.cc:492] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19: Bootstrap starting.
I20260925 15:04:20.558634 25726 tablet_bootstrap.cc:492] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585: Bootstrap starting.
I20260925 15:04:20.561115 25724 tablet_bootstrap.cc:654] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:20.561415 25725 tablet_bootstrap.cc:492] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab: Bootstrap starting.
I20260925 15:04:20.561857 25726 tablet_bootstrap.cc:654] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:20.562557 25724 log.cc:824] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:20.563009 25726 log.cc:824] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:20.564652 25725 tablet_bootstrap.cc:654] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:20.565199 25724 tablet_bootstrap.cc:492] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19: No bootstrap required, opened a new log
I20260925 15:04:20.565436 25724 ts_tablet_manager.cc:1397] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 15:04:20.565744 25725 log.cc:824] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:20.568387 25726 tablet_bootstrap.cc:492] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585: No bootstrap required, opened a new log
I20260925 15:04:20.568538 25725 tablet_bootstrap.cc:492] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab: No bootstrap required, opened a new log
I20260925 15:04:20.568645 25726 ts_tablet_manager.cc:1397] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 15:04:20.568776 25725 ts_tablet_manager.cc:1397] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:04:20.575275 25724 raft_consensus.cc:348] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } }
I20260925 15:04:20.575747 25724 raft_consensus.cc:374] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:20.575845 25724 raft_consensus.cc:729] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc66b1554ef4cb2b5195429e74b8e19, State: Initialized, Role: FOLLOWER
I20260925 15:04:20.575932 25726 raft_consensus.cc:348] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } }
I20260925 15:04:20.576350 25726 raft_consensus.cc:374] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:20.576443 25726 raft_consensus.cc:729] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab5582485d84191bde61fd1c9a18585, State: Initialized, Role: FOLLOWER
I20260925 15:04:20.576491 25724 consensus_queue.cc:260] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } }
I20260925 15:04:20.577005 25726 consensus_queue.cc:260] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } }
I20260925 15:04:20.577628 25566 heartbeater.cc:503] Master 127.22.236.62:38627 was elected leader, sending a full tablet report...
I20260925 15:04:20.578248 25724 ts_tablet_manager.cc:1428] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 15:04:20.578476 25725 raft_consensus.cc:348] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } }
I20260925 15:04:20.578884 25725 raft_consensus.cc:374] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:20.578966 25725 raft_consensus.cc:729] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9d90c6ae3548c5b66bef265e5b34ab, State: Initialized, Role: FOLLOWER
I20260925 15:04:20.579237 25705 heartbeater.cc:503] Master 127.22.236.62:38627 was elected leader, sending a full tablet report...
I20260925 15:04:20.579603 25725 consensus_queue.cc:260] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } }
I20260925 15:04:20.579945 25726 ts_tablet_manager.cc:1428] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260925 15:04:20.581627 25725 ts_tablet_manager.cc:1428] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
W20260925 15:04:20.700512 25706 tablet.cc:2369] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:04:20.732928 25567 tablet.cc:2369] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:04:20.760835 25428 tablet.cc:2369] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:04:20.763182 25730 raft_consensus.cc:482] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:04:20.763437 25730 raft_consensus.cc:504] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } }
I20260925 15:04:20.764995 25730 leader_election.cc:288] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033), 1f9d90c6ae3548c5b66bef265e5b34ab (127.22.236.1:34715)
I20260925 15:04:20.771934 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.765379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:41033 (local address 127.22.236.2:49345)
0925 15:04:20.766072 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:20.766090 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:20.766230 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:20.766725 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:20.766730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:20.766759 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:20.767056 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.767064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.768229 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.768234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.770072 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.770086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.771318 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.771323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.771439 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.771462 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:04:20.771531 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:04:20.771614 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":21,"thread_start_us":388,"threads_started":1}
I20260925 15:04:20.771991 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.765451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:49345 (local address 127.22.236.3:41033)
0925 15:04:20.765616 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:20.765621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:20.766169 (+ 548us) server_negotiation.cc:407] Connection header received
0925 15:04:20.766378 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:20.766383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:20.766454 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:20.766553 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:20.767180 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.768115 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.770225 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.771129 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.771290 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.771369 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:20.771577 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 15:04:20.771699 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:04:20.772742 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.766410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:35505 (local address 127.22.236.1:34715)
0925 15:04:20.766636 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:20.766641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:20.767027 (+ 386us) server_negotiation.cc:407] Connection header received
0925 15:04:20.767242 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:20.767247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:20.767302 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:20.767364 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:20.767909 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.768802 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.770788 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.771850 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.771994 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.772056 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:20.772398 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 15:04:20.772516 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:04:20.772790 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.766216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:34715 (local address 127.22.236.2:35505)
0925 15:04:20.766943 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:20.766956 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:20.767079 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:20.767400 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:20.767405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:20.767429 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:20.767758 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.767769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.768934 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.768939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.770636 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.770646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.772054 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.772061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.772253 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.772285 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:04:20.772356 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:04:20.772465 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":5,"thread_start_us":388,"threads_started":1}
I20260925 15:04:20.772954 25655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2981cc9a2c4fa783e208e20bb830e0" candidate_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab5582485d84191bde61fd1c9a18585" is_pre_election: true
I20260925 15:04:20.773511 25655 raft_consensus.cc:2454] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc66b1554ef4cb2b5195429e74b8e19 in term 0.
I20260925 15:04:20.773612 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2981cc9a2c4fa783e208e20bb830e0" candidate_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" is_pre_election: true
I20260925 15:04:20.774078 25382 raft_consensus.cc:2454] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc66b1554ef4cb2b5195429e74b8e19 in term 0.
I20260925 15:04:20.774158 25449 leader_election.cc:302] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab5582485d84191bde61fd1c9a18585, 4dc66b1554ef4cb2b5195429e74b8e19; no voters:
I20260925 15:04:20.774535 25730 raft_consensus.cc:2781] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:04:20.774731 25730 raft_consensus.cc:482] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:04:20.774844 25730 raft_consensus.cc:3037] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:20.776966 25730 raft_consensus.cc:504] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } }
I20260925 15:04:20.777525 25730 leader_election.cc:288] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [CANDIDATE]: Term 1 election: Requested vote from peers 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033), 1f9d90c6ae3548c5b66bef265e5b34ab (127.22.236.1:34715)
I20260925 15:04:20.777894 25655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2981cc9a2c4fa783e208e20bb830e0" candidate_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab5582485d84191bde61fd1c9a18585"
I20260925 15:04:20.777984 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2981cc9a2c4fa783e208e20bb830e0" candidate_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab"
I20260925 15:04:20.778100 25655 raft_consensus.cc:3037] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:20.778162 25382 raft_consensus.cc:3037] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:20.781198 25382 raft_consensus.cc:2454] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc66b1554ef4cb2b5195429e74b8e19 in term 1.
I20260925 15:04:20.781198 25655 raft_consensus.cc:2454] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc66b1554ef4cb2b5195429e74b8e19 in term 1.
I20260925 15:04:20.781582 25449 leader_election.cc:302] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab5582485d84191bde61fd1c9a18585, 4dc66b1554ef4cb2b5195429e74b8e19; no voters:
I20260925 15:04:20.781848 25730 raft_consensus.cc:2781] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:20.782287 25730 raft_consensus.cc:686] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 LEADER]: Becoming Leader. State: Replica: 4dc66b1554ef4cb2b5195429e74b8e19, State: Running, Role: LEADER
I20260925 15:04:20.782775 25730 consensus_queue.cc:237] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } }
I20260925 15:04:20.787047 25219 catalog_manager.cc:5495] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dc66b1554ef4cb2b5195429e74b8e19 (127.22.236.2). New cstate: current_term: 1 leader_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:04:20 dist-test-slave-nsrc krb5kdc[25175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348658, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:20.855059 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.842648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.22.236.62:38627)
0925 15:04:20.842838 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:20.842842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:20.843326 (+ 484us) server_negotiation.cc:407] Connection header received
0925 15:04:20.843494 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:20.843497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:20.843550 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:20.843625 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:20.844797 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.845496 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.846702 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.847104 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.847205 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.850359 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:20.850363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:20.850366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:20.850393 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:20.852681 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:20.853299 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:20.853304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:20.853307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:20.853412 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:20.853732 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:20.853737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:20.853741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:20.853986 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:20.854100 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:20.854478 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 15:04:20.854761 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":90}
I20260925 15:04:20.855080 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:20.842668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38627 (local address 127.0.0.1:56702)
0925 15:04:20.843224 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:20.843243 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:20.843384 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:20.843666 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:20.843670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:20.844376 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:20.844647 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.844657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.845634 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.845637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.846574 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:20.846585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:20.847281 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:20.847287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:20.847424 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:20.848033 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:20.848050 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:20.850183 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:20.852791 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:20.852803 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:20.852811 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:20.853189 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:20.853506 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:20.853511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:20.853515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:20.853641 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:20.854100 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:20.854108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:20.854250 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 15:04:20.854380 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 15:04:20.854631 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":307,"threads_started":1}
I20260925 15:04:20.860013 25219 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:56702:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:04:20.861044 25219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:04:20.867187 25357 tablet_service.cc:1467] Processing CreateTablet for tablet 38afddadc404488cb32ab20115a0ebe5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2c5e62c16e7648aeb2ad2a506676a989]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:04:20.867288 25496 tablet_service.cc:1467] Processing CreateTablet for tablet 38afddadc404488cb32ab20115a0ebe5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2c5e62c16e7648aeb2ad2a506676a989]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:04:20.867621 25635 tablet_service.cc:1467] Processing CreateTablet for tablet 38afddadc404488cb32ab20115a0ebe5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2c5e62c16e7648aeb2ad2a506676a989]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:04:20.868209 25496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38afddadc404488cb32ab20115a0ebe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:20.868227 25357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38afddadc404488cb32ab20115a0ebe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:20.868424 25635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38afddadc404488cb32ab20115a0ebe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:20.873644 25724 tablet_bootstrap.cc:492] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19: Bootstrap starting.
I20260925 15:04:20.874838 25725 tablet_bootstrap.cc:492] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab: Bootstrap starting.
I20260925 15:04:20.875066 25724 tablet_bootstrap.cc:654] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:20.876389 25725 tablet_bootstrap.cc:654] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:20.877346 25724 tablet_bootstrap.cc:492] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19: No bootstrap required, opened a new log
I20260925 15:04:20.877372 25726 tablet_bootstrap.cc:492] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585: Bootstrap starting.
I20260925 15:04:20.877473 25724 ts_tablet_manager.cc:1397] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:04:20.878051 25724 raft_consensus.cc:348] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:20.878237 25724 raft_consensus.cc:374] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:20.878326 25724 raft_consensus.cc:729] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc66b1554ef4cb2b5195429e74b8e19, State: Initialized, Role: FOLLOWER
I20260925 15:04:20.878602 25726 tablet_bootstrap.cc:654] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:20.878535 25724 consensus_queue.cc:260] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:20.878898 25730 tablet_replica.cc:439] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } }
I20260925 15:04:20.879060 25724 ts_tablet_manager.cc:1428] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 15:04:20.879195 25730 tablet_replica.cc:442] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:04:20.880648 25725 tablet_bootstrap.cc:492] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab: No bootstrap required, opened a new log
I20260925 15:04:20.880851 25725 ts_tablet_manager.cc:1397] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260925 15:04:20.881618 25725 raft_consensus.cc:348] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:20.881875 25725 raft_consensus.cc:374] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:20.881973 25725 raft_consensus.cc:729] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9d90c6ae3548c5b66bef265e5b34ab, State: Initialized, Role: FOLLOWER
I20260925 15:04:20.882207 25725 consensus_queue.cc:260] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:20.882794 25725 ts_tablet_manager.cc:1428] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 15:04:20.882915 25732 tablet_replica.cc:439] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } }
I20260925 15:04:20.883332 25732 tablet_replica.cc:442] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:04:20.890405 25726 tablet_bootstrap.cc:492] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585: No bootstrap required, opened a new log
I20260925 15:04:20.890542 25726 ts_tablet_manager.cc:1397] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260925 15:04:20.891135 25726 raft_consensus.cc:348] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:20.891320 25726 raft_consensus.cc:374] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:20.891409 25726 raft_consensus.cc:729] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab5582485d84191bde61fd1c9a18585, State: Initialized, Role: FOLLOWER
I20260925 15:04:20.891621 25726 consensus_queue.cc:260] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:20.892122 25726 ts_tablet_manager.cc:1428] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 15:04:20.892077 25731 tablet_replica.cc:439] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } }
I20260925 15:04:20.892494 25731 tablet_replica.cc:442] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:04:21.027776 25732 raft_consensus.cc:482] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:04:21.027966 25732 raft_consensus.cc:504] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:21.029407 25732 leader_election.cc:288] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dc66b1554ef4cb2b5195429e74b8e19 (127.22.236.2:41355), 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033)
I20260925 15:04:21.035148 25730 raft_consensus.cc:482] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:04:21.035326 25730 raft_consensus.cc:504] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:21.035992 25730 leader_election.cc:288] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9d90c6ae3548c5b66bef265e5b34ab (127.22.236.1:34715), 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033)
I20260925 15:04:21.037124 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38afddadc404488cb32ab20115a0ebe5" candidate_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" is_pre_election: true
I20260925 15:04:21.037376 25382 raft_consensus.cc:2454] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc66b1554ef4cb2b5195429e74b8e19 in term 0.
I20260925 15:04:21.037542 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:21.029785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:60751 (local address 127.22.236.2:41355)
0925 15:04:21.029956 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:21.029961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:21.030536 (+ 575us) server_negotiation.cc:407] Connection header received
0925 15:04:21.030724 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:21.030730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:21.030805 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:21.030899 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:21.031638 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.032731 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.035518 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.036696 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.036842 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:21.036923 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:21.037160 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 15:04:21.037296 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:04:21.037665 25655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38afddadc404488cb32ab20115a0ebe5" candidate_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab5582485d84191bde61fd1c9a18585" is_pre_election: true
I20260925 15:04:21.037839 25449 leader_election.cc:302] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9d90c6ae3548c5b66bef265e5b34ab, 4dc66b1554ef4cb2b5195429e74b8e19; no voters:
I20260925 15:04:21.037916 25655 raft_consensus.cc:2454] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc66b1554ef4cb2b5195429e74b8e19 in term 0.
I20260925 15:04:21.038151 25730 raft_consensus.cc:2781] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:04:21.038244 25730 raft_consensus.cc:482] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:04:21.038265 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:21.029636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41355 (local address 127.22.236.1:60751)
0925 15:04:21.030421 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:21.030440 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:21.030590 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:21.030924 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:21.030929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:21.030973 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:21.031468 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:21.031476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.032890 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.032895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:21.035361 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:21.035377 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.036883 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.036888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:21.037003 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:21.037032 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:04:21.037118 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:04:21.037239 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":144,"spinlock_wait_cycles":17664,"thread_start_us":474,"threads_started":1}
I20260925 15:04:21.038349 25730 raft_consensus.cc:3037] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:21.038954 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:21.029773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:40367 (local address 127.22.236.3:41033)
0925 15:04:21.029941 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:21.029946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:21.030571 (+ 625us) server_negotiation.cc:407] Connection header received
0925 15:04:21.030724 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:21.030730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:21.030805 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:21.030925 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:21.031638 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.032731 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.036213 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.037292 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.037443 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:21.037507 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:21.038606 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0925 15:04:21.038725 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:04:21.039247 25516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38afddadc404488cb32ab20115a0ebe5" candidate_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" is_pre_election: true
I20260925 15:04:21.039567 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:21.029705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:41033 (local address 127.22.236.1:40367)
0925 15:04:21.030478 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:21.030491 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:21.030598 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:21.030927 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:21.030931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:21.030974 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:21.031461 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:21.031473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.032889 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.032895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:21.035368 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:21.035377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.038236 (+ 2859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.038243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:21.038437 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:21.038466 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:04:21.039095 (+ 629us) client_negotiation.cc:240] Negotiation successful
0925 15:04:21.039222 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":157,"spinlock_wait_cycles":33408,"thread_start_us":466,"threads_started":1}
I20260925 15:04:21.040112 25655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38afddadc404488cb32ab20115a0ebe5" candidate_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab5582485d84191bde61fd1c9a18585" is_pre_election: true
I20260925 15:04:21.040337 25655 raft_consensus.cc:2454] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9d90c6ae3548c5b66bef265e5b34ab in term 0.
I20260925 15:04:21.040762 25309 leader_election.cc:302] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab5582485d84191bde61fd1c9a18585, 1f9d90c6ae3548c5b66bef265e5b34ab; no voters:
I20260925 15:04:21.041152 25732 raft_consensus.cc:2781] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:04:21.041189 25730 raft_consensus.cc:504] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:21.041291 25732 raft_consensus.cc:482] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:04:21.041388 25732 raft_consensus.cc:3037] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:21.041615 25516 raft_consensus.cc:2379] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f9d90c6ae3548c5b66bef265e5b34ab in current term 1: Already voted for candidate 4dc66b1554ef4cb2b5195429e74b8e19 in this term.
I20260925 15:04:21.041739 25730 leader_election.cc:288] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [CANDIDATE]: Term 1 election: Requested vote from peers 1f9d90c6ae3548c5b66bef265e5b34ab (127.22.236.1:34715), 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033)
I20260925 15:04:21.042189 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38afddadc404488cb32ab20115a0ebe5" candidate_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab"
I20260925 15:04:21.042472 25655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38afddadc404488cb32ab20115a0ebe5" candidate_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab5582485d84191bde61fd1c9a18585"
I20260925 15:04:21.042652 25655 raft_consensus.cc:3037] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:21.043926 25732 raft_consensus.cc:504] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:21.044508 25732 leader_election.cc:288] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [CANDIDATE]: Term 1 election: Requested vote from peers 4dc66b1554ef4cb2b5195429e74b8e19 (127.22.236.2:41355), 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033)
I20260925 15:04:21.044512 25382 raft_consensus.cc:2379] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4dc66b1554ef4cb2b5195429e74b8e19 in current term 1: Already voted for candidate 1f9d90c6ae3548c5b66bef265e5b34ab in this term.
I20260925 15:04:21.044792 25655 raft_consensus.cc:2454] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc66b1554ef4cb2b5195429e74b8e19 in term 1.
I20260925 15:04:21.044931 25516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38afddadc404488cb32ab20115a0ebe5" candidate_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc66b1554ef4cb2b5195429e74b8e19"
I20260925 15:04:21.045142 25516 raft_consensus.cc:2379] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f9d90c6ae3548c5b66bef265e5b34ab in current term 1: Already voted for candidate 4dc66b1554ef4cb2b5195429e74b8e19 in this term.
I20260925 15:04:21.045243 25449 leader_election.cc:302] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab5582485d84191bde61fd1c9a18585, 4dc66b1554ef4cb2b5195429e74b8e19; no voters:
I20260925 15:04:21.045398 25655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38afddadc404488cb32ab20115a0ebe5" candidate_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab5582485d84191bde61fd1c9a18585"
I20260925 15:04:21.045629 25655 raft_consensus.cc:2379] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f9d90c6ae3548c5b66bef265e5b34ab in current term 1: Already voted for candidate 4dc66b1554ef4cb2b5195429e74b8e19 in this term.
I20260925 15:04:21.046013 25730 raft_consensus.cc:2781] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:21.046180 25730 raft_consensus.cc:686] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 LEADER]: Becoming Leader. State: Replica: 4dc66b1554ef4cb2b5195429e74b8e19, State: Running, Role: LEADER
I20260925 15:04:21.046160 25309 leader_election.cc:302] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f9d90c6ae3548c5b66bef265e5b34ab; no voters: 0ab5582485d84191bde61fd1c9a18585, 4dc66b1554ef4cb2b5195429e74b8e19
I20260925 15:04:21.046392 25732 raft_consensus.cc:2726] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 15:04:21.046394 25730 consensus_queue.cc:237] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:21.047979 25735 tablet_replica.cc:439] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19: TxnStatusTablet state changed. Reason: New leader 4dc66b1554ef4cb2b5195429e74b8e19. Latest consensus state: current_term: 1 leader_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } }
I20260925 15:04:21.048352 25735 tablet_replica.cc:442] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:04:21.048944 25749 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:04:21.049175 25749 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:04:21.049232 25220 catalog_manager.cc:5495] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dc66b1554ef4cb2b5195429e74b8e19 (127.22.236.2). New cstate: current_term: 1 leader_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } health_report { overall_health: UNKNOWN } } }
I20260925 15:04:21.071121 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:21.065157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41355 (local address 127.0.0.1:50738)
0925 15:04:21.065352 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:21.065380 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:21.065485 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:21.065944 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:21.065947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:21.065957 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:04:21.066101 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:21.066108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.067515 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.067519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:21.068613 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:21.068624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.069542 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.069546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:21.069682 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:21.069732 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:04:21.070680 (+ 948us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:04:21.070712 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:04:21.070775 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 15:04:21.070906 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 15:04:21.071210 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:21.065294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.22.236.2:41355)
0925 15:04:21.065474 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:21.065483 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:21.065503 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:04:21.065662 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:21.065668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:21.065786 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:21.065929 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:21.066222 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.067388 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.068758 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.069411 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.069532 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:21.069902 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:21.070582 (+ 680us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:21.070746 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:21.070796 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 15:04:21.070874 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:04:21.078078 25382 raft_consensus.cc:1261] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Refusing update from remote peer 4dc66b1554ef4cb2b5195429e74b8e19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:21.078188 25655 raft_consensus.cc:1261] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Refusing update from remote peer 4dc66b1554ef4cb2b5195429e74b8e19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:21.078794 25735 consensus_queue.cc:1035] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:21.079138 25730 consensus_queue.cc:1035] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:21.094849 25731 tablet_replica.cc:439] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585: TxnStatusTablet state changed. Reason: New leader 4dc66b1554ef4cb2b5195429e74b8e19. Latest consensus state: current_term: 1 leader_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } }
I20260925 15:04:21.095130 25731 tablet_replica.cc:442] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:04:21.095795 25732 tablet_replica.cc:439] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab: TxnStatusTablet state changed. Reason: New leader 4dc66b1554ef4cb2b5195429e74b8e19. Latest consensus state: current_term: 1 leader_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } }
I20260925 15:04:21.096096 25732 tablet_replica.cc:442] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:04:21.098922 25735 tablet_replica.cc:439] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } }
I20260925 15:04:21.099174 25735 tablet_replica.cc:442] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:04:21.100966 25730 tablet_replica.cc:439] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } }
I20260925 15:04:21.101234 25730 tablet_replica.cc:442] T 38afddadc404488cb32ab20115a0ebe5 P 4dc66b1554ef4cb2b5195429e74b8e19: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:04:21.110525 25732 tablet_replica.cc:439] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } }
I20260925 15:04:21.110788 25732 tablet_replica.cc:442] T 38afddadc404488cb32ab20115a0ebe5 P 1f9d90c6ae3548c5b66bef265e5b34ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:04:21.112026 25731 tablet_replica.cc:439] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } } }
I20260925 15:04:21.112283 25731 tablet_replica.cc:442] T 38afddadc404488cb32ab20115a0ebe5 P 0ab5582485d84191bde61fd1c9a18585: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:04:21.113112 25754 mvcc.cc:205] Tried to move back new op lower bound from 7333268115767209984 to 7333268115648290816. Current Snapshot: MvccSnapshot[applied={T|T < 7333268115767209984 or (T in {7333268115767209984})}]
I20260925 15:04:21.118568 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:21.113598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.22.236.2:41355)
0925 15:04:21.113807 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:21.113813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:21.114159 (+ 346us) server_negotiation.cc:407] Connection header received
0925 15:04:21.114354 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:21.114359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:21.114436 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:21.114507 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:21.114895 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.115641 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.116983 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.117431 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.117542 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:21.117929 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:21.118025 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:21.118105 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:21.118241 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 15:04:21.118312 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 15:04:21.134057 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:21.128936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:37005 (local address 127.22.236.2:41355)
0925 15:04:21.129108 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:21.129113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:21.129494 (+ 381us) server_negotiation.cc:407] Connection header received
0925 15:04:21.129650 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:21.129655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:21.129705 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:21.129772 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:21.130254 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.131011 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.132407 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.132888 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.132991 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:21.133390 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:21.133494 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:21.133567 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:21.133685 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 15:04:21.133745 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:04:21.134135 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:21.128885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41355 (local address 127.22.236.2:37005)
0925 15:04:21.129404 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:21.129419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:21.129545 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:21.129789 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:21.129793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:21.129803 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:04:21.130128 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:21.130136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.131122 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.131128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:21.132280 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:21.132291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.133053 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.133059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:21.133169 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:21.133219 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:04:21.133584 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:04:21.133593 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:04:21.133659 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:04:21.133778 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":248,"threads_started":1}
I20260925 15:04:21.142736 25381 raft_consensus.cc:1261] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Refusing update from remote peer 4dc66b1554ef4cb2b5195429e74b8e19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:21.142799 25655 raft_consensus.cc:1261] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Refusing update from remote peer 4dc66b1554ef4cb2b5195429e74b8e19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:21.143240 25730 consensus_queue.cc:1035] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:21.143493 25735 consensus_queue.cc:1035] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:21.147310 25752 mvcc.cc:205] Tried to move back new op lower bound from 7333268116036337664 to 7333268114570022912. Current Snapshot: MvccSnapshot[applied={T|T < 7333268116036337664}]
I20260925 15:04:21.152371 25753 mvcc.cc:205] Tried to move back new op lower bound from 7333268116036337664 to 7333268114570022912. Current Snapshot: MvccSnapshot[applied={T|T < 7333268116036337664}]
I20260925 15:04:21.216259 25219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:21.216485 25219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:21.219630 25219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182: Sending DeleteTablet for 3 replicas of tablet 1e2981cc9a2c4fa783e208e20bb830e0
I20260925 15:04:21.220746 25635 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e2981cc9a2c4fa783e208e20bb830e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:21 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:40006
I20260925 15:04:21.220839 25357 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e2981cc9a2c4fa783e208e20bb830e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:21 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:38414
I20260925 15:04:21.221550 25769 tablet_replica.cc:330] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585: stopping tablet replica
I20260925 15:04:21.221590 25495 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e2981cc9a2c4fa783e208e20bb830e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:21 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:50722
I20260925 15:04:21.222110 25769 raft_consensus.cc:2229] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:21.222536 25769 raft_consensus.cc:2258] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:21.222440 25219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:04:21.223068 25219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:04:21.224900 25769 ts_tablet_manager.cc:1905] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:21.231629 25635 tablet_service.cc:1467] Processing CreateTablet for tablet 16ca5a474a6e400185693a314ff2abb6 (DEFAULT_TABLE table=test-table [id=2cad0d58763a4928bc4aac4b571208fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:21.231684 25356 tablet_service.cc:1467] Processing CreateTablet for tablet 16ca5a474a6e400185693a314ff2abb6 (DEFAULT_TABLE table=test-table [id=2cad0d58763a4928bc4aac4b571208fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:21.232033 25635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ca5a474a6e400185693a314ff2abb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:21.232159 25356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ca5a474a6e400185693a314ff2abb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:21.233193 25770 tablet_replica.cc:330] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab: stopping tablet replica
I20260925 15:04:21.233229 25769 ts_tablet_manager.cc:1918] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:04:21.233474 25769 log.cc:1196] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/1e2981cc9a2c4fa783e208e20bb830e0
I20260925 15:04:21.233649 25770 raft_consensus.cc:2229] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:21.234022 25770 raft_consensus.cc:2258] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:21.234172 25769 ts_tablet_manager.cc:1939] T 1e2981cc9a2c4fa783e208e20bb830e0 P 0ab5582485d84191bde61fd1c9a18585: Deleting consensus metadata
I20260925 15:04:21.236470 25206 catalog_manager.cc:4841] TS 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033): tablet 1e2981cc9a2c4fa783e208e20bb830e0 (table test-table [id=4604f5859fbf4c078adb12049e0b3c14]) successfully deleted
I20260925 15:04:21.238251 25726 tablet_bootstrap.cc:492] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585: Bootstrap starting.
I20260925 15:04:21.238862 25496 tablet_service.cc:1467] Processing CreateTablet for tablet 16ca5a474a6e400185693a314ff2abb6 (DEFAULT_TABLE table=test-table [id=2cad0d58763a4928bc4aac4b571208fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:21.239251 25496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ca5a474a6e400185693a314ff2abb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:21.239926 25726 tablet_bootstrap.cc:654] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:21.240988 25725 tablet_bootstrap.cc:492] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab: Bootstrap starting.
I20260925 15:04:21.242434 25725 tablet_bootstrap.cc:654] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:21.251602 25770 ts_tablet_manager.cc:1905] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:21.253378 25724 tablet_bootstrap.cc:492] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19: Bootstrap starting.
I20260925 15:04:21.253973 25771 tablet_replica.cc:330] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19: stopping tablet replica
I20260925 15:04:21.254415 25771 raft_consensus.cc:2229] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:04:21.254961 25724 tablet_bootstrap.cc:654] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:21.255018 25771 raft_consensus.cc:2258] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:21.257078 25771 ts_tablet_manager.cc:1905] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:21.262346 25724 tablet_bootstrap.cc:492] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19: No bootstrap required, opened a new log
I20260925 15:04:21.262482 25770 ts_tablet_manager.cc:1918] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:04:21.262521 25724 ts_tablet_manager.cc:1397] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260925 15:04:21.262632 25770 log.cc:1196] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/1e2981cc9a2c4fa783e208e20bb830e0
I20260925 15:04:21.263088 25770 ts_tablet_manager.cc:1939] T 1e2981cc9a2c4fa783e208e20bb830e0 P 1f9d90c6ae3548c5b66bef265e5b34ab: Deleting consensus metadata
I20260925 15:04:21.263329 25724 raft_consensus.cc:348] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:21.263584 25724 raft_consensus.cc:374] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:21.263664 25724 raft_consensus.cc:729] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc66b1554ef4cb2b5195429e74b8e19, State: Initialized, Role: FOLLOWER
I20260925 15:04:21.263969 25724 consensus_queue.cc:260] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:21.265506 25771 ts_tablet_manager.cc:1918] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:04:21.265700 25771 log.cc:1196] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/1e2981cc9a2c4fa783e208e20bb830e0
I20260925 15:04:21.266292 25771 ts_tablet_manager.cc:1939] T 1e2981cc9a2c4fa783e208e20bb830e0 P 4dc66b1554ef4cb2b5195429e74b8e19: Deleting consensus metadata
I20260925 15:04:21.267010 25724 ts_tablet_manager.cc:1428] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:04:21.267175 25206 catalog_manager.cc:4841] TS 1f9d90c6ae3548c5b66bef265e5b34ab (127.22.236.1:34715): tablet 1e2981cc9a2c4fa783e208e20bb830e0 (table test-table [id=4604f5859fbf4c078adb12049e0b3c14]) successfully deleted
I20260925 15:04:21.268006 25205 catalog_manager.cc:4841] TS 4dc66b1554ef4cb2b5195429e74b8e19 (127.22.236.2:41355): tablet 1e2981cc9a2c4fa783e208e20bb830e0 (table test-table [id=4604f5859fbf4c078adb12049e0b3c14]) successfully deleted
I20260925 15:04:21.270723 25725 tablet_bootstrap.cc:492] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab: No bootstrap required, opened a new log
I20260925 15:04:21.270884 25725 ts_tablet_manager.cc:1397] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.004s
I20260925 15:04:21.271487 25726 tablet_bootstrap.cc:492] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585: No bootstrap required, opened a new log
I20260925 15:04:21.271658 25726 ts_tablet_manager.cc:1397] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.020s
I20260925 15:04:21.271586 25725 raft_consensus.cc:348] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:21.271806 25725 raft_consensus.cc:374] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:21.271885 25725 raft_consensus.cc:729] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9d90c6ae3548c5b66bef265e5b34ab, State: Initialized, Role: FOLLOWER
I20260925 15:04:21.272105 25725 consensus_queue.cc:260] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:21.272377 25726 raft_consensus.cc:348] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:21.272596 25726 raft_consensus.cc:374] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:21.272661 25726 raft_consensus.cc:729] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab5582485d84191bde61fd1c9a18585, State: Initialized, Role: FOLLOWER
I20260925 15:04:21.272693 25725 ts_tablet_manager.cc:1428] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:04:21.272920 25726 consensus_queue.cc:260] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:21.274317 25726 ts_tablet_manager.cc:1428] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:04:21.549038 25732 raft_consensus.cc:482] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:04:21.549203 25732 raft_consensus.cc:504] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:21.549758 25732 leader_election.cc:288] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dc66b1554ef4cb2b5195429e74b8e19 (127.22.236.2:41355), 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033)
I20260925 15:04:21.550271 25655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca5a474a6e400185693a314ff2abb6" candidate_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab5582485d84191bde61fd1c9a18585" is_pre_election: true
I20260925 15:04:21.550323 25516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca5a474a6e400185693a314ff2abb6" candidate_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" is_pre_election: true
I20260925 15:04:21.550474 25655 raft_consensus.cc:2454] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9d90c6ae3548c5b66bef265e5b34ab in term 0.
I20260925 15:04:21.550611 25516 raft_consensus.cc:2454] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9d90c6ae3548c5b66bef265e5b34ab in term 0.
I20260925 15:04:21.550863 25309 leader_election.cc:302] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab5582485d84191bde61fd1c9a18585, 1f9d90c6ae3548c5b66bef265e5b34ab; no voters:
I20260925 15:04:21.551121 25732 raft_consensus.cc:2781] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:04:21.551213 25732 raft_consensus.cc:482] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:04:21.551303 25732 raft_consensus.cc:3037] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:21.553344 25732 raft_consensus.cc:504] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:21.553851 25732 leader_election.cc:288] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [CANDIDATE]: Term 1 election: Requested vote from peers 4dc66b1554ef4cb2b5195429e74b8e19 (127.22.236.2:41355), 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033)
I20260925 15:04:21.554291 25655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca5a474a6e400185693a314ff2abb6" candidate_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab5582485d84191bde61fd1c9a18585"
I20260925 15:04:21.554453 25655 raft_consensus.cc:3037] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:21.556310 25655 raft_consensus.cc:2454] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9d90c6ae3548c5b66bef265e5b34ab in term 1.
I20260925 15:04:21.556691 25309 leader_election.cc:302] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab5582485d84191bde61fd1c9a18585, 1f9d90c6ae3548c5b66bef265e5b34ab; no voters:
I20260925 15:04:21.556666 25516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca5a474a6e400185693a314ff2abb6" candidate_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc66b1554ef4cb2b5195429e74b8e19"
I20260925 15:04:21.556862 25516 raft_consensus.cc:3037] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:21.559047 25516 raft_consensus.cc:2454] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9d90c6ae3548c5b66bef265e5b34ab in term 1.
I20260925 15:04:21.559291 25732 raft_consensus.cc:2781] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:21.559748 25732 raft_consensus.cc:686] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 LEADER]: Becoming Leader. State: Replica: 1f9d90c6ae3548c5b66bef265e5b34ab, State: Running, Role: LEADER
I20260925 15:04:21.560088 25732 consensus_queue.cc:237] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:21.563432 25220 catalog_manager.cc:5495] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9d90c6ae3548c5b66bef265e5b34ab (127.22.236.1). New cstate: current_term: 1 leader_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } health_report { overall_health: UNKNOWN } } }
I20260925 15:04:21.640089 25779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:21.634760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.22.236.1:34715)
0925 15:04:21.635196 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:21.635202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:21.635224 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:04:21.635329 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:21.635333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:21.635416 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:21.635561 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:21.635996 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.636753 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.638027 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.638412 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.638513 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:21.638927 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:21.639506 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:21.639635 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:21.639738 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 15:04:21.639814 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":256,"threads_started":1}
I20260925 15:04:21.654860 25782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:21.649457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41355 (local address 127.22.236.1:59899)
0925 15:04:21.649974 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:21.649994 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:21.650118 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:21.650490 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:21.650495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:21.650506 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:04:21.650734 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:21.650742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.651736 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.651740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:21.652944 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:21.652956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.653694 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.653700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:21.653824 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:21.653878 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:04:21.654292 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:04:21.654303 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:04:21.654381 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:04:21.654516 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
I20260925 15:04:21.654832 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:21.649602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:59899 (local address 127.22.236.2:41355)
0925 15:04:21.650211 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:21.650217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:21.650233 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:04:21.650291 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:21.650294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:21.650351 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:21.650449 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:21.650862 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.651611 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.653074 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.653541 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.653651 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:21.654072 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:21.654183 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:21.654270 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:21.654414 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 15:04:21.654495 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":343,"threads_started":1}
I20260925 15:04:21.665037 25655 raft_consensus.cc:1261] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Refusing update from remote peer 1f9d90c6ae3548c5b66bef265e5b34ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:21.665376 25516 raft_consensus.cc:1261] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Refusing update from remote peer 1f9d90c6ae3548c5b66bef265e5b34ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:21.665779 25775 consensus_queue.cc:1035] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:21.666175 25732 consensus_queue.cc:1035] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:21.699234 25779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:21.693266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:60805 (local address 127.22.236.1:34715)
0925 15:04:21.693429 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:21.693435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:21.693602 (+ 167us) server_negotiation.cc:407] Connection header received
0925 15:04:21.693826 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:21.693831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:21.693915 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:21.693991 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:21.694685 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.695729 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.697688 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.698089 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.698195 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:21.698591 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:21.698688 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:21.698757 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:21.698898 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 15:04:21.698972 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:04:21.699649 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:21.692917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:34715 (local address 127.22.236.2:60805)
0925 15:04:21.693500 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:21.693528 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:21.693697 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:21.694160 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:21.694165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:21.694177 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:04:21.694514 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:21.694526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.695865 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.695870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:21.697509 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:21.697527 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:21.698240 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:21.698246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:21.698406 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:21.698451 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:04:21.698780 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:04:21.698790 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:04:21.698859 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:04:21.699003 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":529,"threads_started":1}
W20260925 15:04:21.700706 25754 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 15:04:23.304687 25220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:23.304917 25220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:23.308526 25220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182: Sending DeleteTablet for 3 replicas of tablet 16ca5a474a6e400185693a314ff2abb6
I20260925 15:04:23.309175 25356 tablet_service.cc:1514] Processing DeleteTablet for tablet 16ca5a474a6e400185693a314ff2abb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:23 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:38414
I20260925 15:04:23.309247 25495 tablet_service.cc:1514] Processing DeleteTablet for tablet 16ca5a474a6e400185693a314ff2abb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:23 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:50722
I20260925 15:04:23.309289 25635 tablet_service.cc:1514] Processing DeleteTablet for tablet 16ca5a474a6e400185693a314ff2abb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:23 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:40006
I20260925 15:04:23.320263 25817 tablet_replica.cc:330] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585: stopping tablet replica
I20260925 15:04:23.320636 25817 raft_consensus.cc:2229] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:23.320969 25817 raft_consensus.cc:2258] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:23.321537 25816 tablet_replica.cc:330] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab: stopping tablet replica
I20260925 15:04:23.321719 25818 tablet_replica.cc:330] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19: stopping tablet replica
I20260925 15:04:23.321897 25818 raft_consensus.cc:2229] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:23.322129 25818 raft_consensus.cc:2258] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:23.322260 25817 ts_tablet_manager.cc:1905] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:23.322451 25816 raft_consensus.cc:2229] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:04:23.323024 25816 raft_consensus.cc:2258] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:23.323056 25818 ts_tablet_manager.cc:1905] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:23.323817 25816 ts_tablet_manager.cc:1905] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:23.329120 25817 ts_tablet_manager.cc:1918] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:04:23.329303 25817 log.cc:1196] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/16ca5a474a6e400185693a314ff2abb6
I20260925 15:04:23.329391 25818 ts_tablet_manager.cc:1918] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:04:23.329566 25818 log.cc:1196] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/16ca5a474a6e400185693a314ff2abb6
I20260925 15:04:23.329741 25817 ts_tablet_manager.cc:1939] T 16ca5a474a6e400185693a314ff2abb6 P 0ab5582485d84191bde61fd1c9a18585: Deleting consensus metadata
I20260925 15:04:23.329974 25816 ts_tablet_manager.cc:1918] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:04:23.330046 25818 ts_tablet_manager.cc:1939] T 16ca5a474a6e400185693a314ff2abb6 P 4dc66b1554ef4cb2b5195429e74b8e19: Deleting consensus metadata
I20260925 15:04:23.330118 25816 log.cc:1196] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/16ca5a474a6e400185693a314ff2abb6
I20260925 15:04:23.330559 25816 ts_tablet_manager.cc:1939] T 16ca5a474a6e400185693a314ff2abb6 P 1f9d90c6ae3548c5b66bef265e5b34ab: Deleting consensus metadata
I20260925 15:04:23.330948 25206 catalog_manager.cc:4841] TS 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033): tablet 16ca5a474a6e400185693a314ff2abb6 (table test-table [id=2cad0d58763a4928bc4aac4b571208fe]) successfully deleted
I20260925 15:04:23.331108 25205 catalog_manager.cc:4841] TS 4dc66b1554ef4cb2b5195429e74b8e19 (127.22.236.2:41355): tablet 16ca5a474a6e400185693a314ff2abb6 (table test-table [id=2cad0d58763a4928bc4aac4b571208fe]) successfully deleted
I20260925 15:04:23.331524 25206 catalog_manager.cc:4841] TS 1f9d90c6ae3548c5b66bef265e5b34ab (127.22.236.1:34715): tablet 16ca5a474a6e400185693a314ff2abb6 (table test-table [id=2cad0d58763a4928bc4aac4b571208fe]) successfully deleted
Sep 25 15:04:23 dist-test-slave-nsrc krb5kdc[25175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348663, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:04:23 dist-test-slave-nsrc krb5kdc[25175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348663, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:23.362962 25826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:23.346949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.22.236.62:38627)
0925 15:04:23.347453 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:23.347458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:23.347502 (+ 44us) server_negotiation.cc:407] Connection header received
0925 15:04:23.347691 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:23.347695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:23.347753 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:23.347839 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:23.349154 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:23.349930 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:23.351114 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:23.351500 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:23.351602 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:23.358319 (+ 6717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:23.358324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:23.358327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:23.358358 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:23.360582 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:23.361189 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:23.361193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:23.361195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:23.361308 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:23.361648 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:23.361653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:23.361656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:23.361928 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:23.362135 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:23.362414 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 15:04:23.362619 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":236,"threads_started":1}
I20260925 15:04:23.366103 25220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:04:23.366658 25220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:04:23.372538 25356 tablet_service.cc:1467] Processing CreateTablet for tablet 3314760a65dd4ae48fe902740f0cb857 (DEFAULT_TABLE table=test-table [id=b2f33bf67f85457b8593d1080d7e7a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:23.372746 25495 tablet_service.cc:1467] Processing CreateTablet for tablet 3314760a65dd4ae48fe902740f0cb857 (DEFAULT_TABLE table=test-table [id=b2f33bf67f85457b8593d1080d7e7a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:23.372805 25635 tablet_service.cc:1467] Processing CreateTablet for tablet 3314760a65dd4ae48fe902740f0cb857 (DEFAULT_TABLE table=test-table [id=b2f33bf67f85457b8593d1080d7e7a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:23.373003 25356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3314760a65dd4ae48fe902740f0cb857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:23.373159 25635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3314760a65dd4ae48fe902740f0cb857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:23.373167 25495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3314760a65dd4ae48fe902740f0cb857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:23.379246 25827 tablet_bootstrap.cc:492] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19: Bootstrap starting.
I20260925 15:04:23.380033 25829 tablet_bootstrap.cc:492] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585: Bootstrap starting.
I20260925 15:04:23.380330 25828 tablet_bootstrap.cc:492] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab: Bootstrap starting.
I20260925 15:04:23.381323 25827 tablet_bootstrap.cc:654] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:23.381707 25829 tablet_bootstrap.cc:654] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:23.381879 25828 tablet_bootstrap.cc:654] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:23.383896 25827 tablet_bootstrap.cc:492] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19: No bootstrap required, opened a new log
I20260925 15:04:23.384006 25828 tablet_bootstrap.cc:492] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab: No bootstrap required, opened a new log
I20260925 15:04:23.384025 25827 ts_tablet_manager.cc:1397] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 15:04:23.384166 25828 ts_tablet_manager.cc:1397] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 15:04:23.384655 25827 raft_consensus.cc:348] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.384848 25827 raft_consensus.cc:374] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:23.384760 25828 raft_consensus.cc:348] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.384938 25828 raft_consensus.cc:374] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:23.384943 25827 raft_consensus.cc:729] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc66b1554ef4cb2b5195429e74b8e19, State: Initialized, Role: FOLLOWER
I20260925 15:04:23.385025 25828 raft_consensus.cc:729] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9d90c6ae3548c5b66bef265e5b34ab, State: Initialized, Role: FOLLOWER
I20260925 15:04:23.385164 25827 consensus_queue.cc:260] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.385236 25828 consensus_queue.cc:260] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.385643 25827 ts_tablet_manager.cc:1428] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 15:04:23.385804 25828 ts_tablet_manager.cc:1428] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:04:23.386843 25829 tablet_bootstrap.cc:492] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585: No bootstrap required, opened a new log
I20260925 15:04:23.386967 25829 ts_tablet_manager.cc:1397] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260925 15:04:23.387658 25829 raft_consensus.cc:348] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.387903 25829 raft_consensus.cc:374] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:23.387998 25829 raft_consensus.cc:729] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab5582485d84191bde61fd1c9a18585, State: Initialized, Role: FOLLOWER
I20260925 15:04:23.388236 25829 consensus_queue.cc:260] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.389149 25829 ts_tablet_manager.cc:1428] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:04:23.539875 25794 raft_consensus.cc:482] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:04:23.540086 25794 raft_consensus.cc:504] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.540695 25794 leader_election.cc:288] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dc66b1554ef4cb2b5195429e74b8e19 (127.22.236.2:41355), 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033)
I20260925 15:04:23.541257 25516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3314760a65dd4ae48fe902740f0cb857" candidate_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" is_pre_election: true
I20260925 15:04:23.541301 25654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3314760a65dd4ae48fe902740f0cb857" candidate_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab5582485d84191bde61fd1c9a18585" is_pre_election: true
I20260925 15:04:23.541477 25516 raft_consensus.cc:2454] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9d90c6ae3548c5b66bef265e5b34ab in term 0.
I20260925 15:04:23.541502 25654 raft_consensus.cc:2454] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9d90c6ae3548c5b66bef265e5b34ab in term 0.
I20260925 15:04:23.541949 25309 leader_election.cc:302] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ab5582485d84191bde61fd1c9a18585, 1f9d90c6ae3548c5b66bef265e5b34ab, 4dc66b1554ef4cb2b5195429e74b8e19; no voters:
I20260925 15:04:23.542192 25794 raft_consensus.cc:2781] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:04:23.542289 25794 raft_consensus.cc:482] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:04:23.542373 25794 raft_consensus.cc:3037] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:23.544327 25794 raft_consensus.cc:504] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.544807 25794 leader_election.cc:288] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [CANDIDATE]: Term 1 election: Requested vote from peers 4dc66b1554ef4cb2b5195429e74b8e19 (127.22.236.2:41355), 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033)
I20260925 15:04:23.545302 25654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3314760a65dd4ae48fe902740f0cb857" candidate_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab5582485d84191bde61fd1c9a18585"
I20260925 15:04:23.545279 25516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3314760a65dd4ae48fe902740f0cb857" candidate_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc66b1554ef4cb2b5195429e74b8e19"
I20260925 15:04:23.545483 25654 raft_consensus.cc:3037] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:23.545490 25516 raft_consensus.cc:3037] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:23.547761 25516 raft_consensus.cc:2454] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9d90c6ae3548c5b66bef265e5b34ab in term 1.
I20260925 15:04:23.547798 25654 raft_consensus.cc:2454] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9d90c6ae3548c5b66bef265e5b34ab in term 1.
I20260925 15:04:23.548167 25309 leader_election.cc:302] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ab5582485d84191bde61fd1c9a18585, 1f9d90c6ae3548c5b66bef265e5b34ab, 4dc66b1554ef4cb2b5195429e74b8e19; no voters:
I20260925 15:04:23.548418 25794 raft_consensus.cc:2781] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:23.548537 25794 raft_consensus.cc:686] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 LEADER]: Becoming Leader. State: Replica: 1f9d90c6ae3548c5b66bef265e5b34ab, State: Running, Role: LEADER
I20260925 15:04:23.548775 25794 consensus_queue.cc:237] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.551311 25219 catalog_manager.cc:5495] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9d90c6ae3548c5b66bef265e5b34ab (127.22.236.1). New cstate: current_term: 1 leader_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } health_report { overall_health: UNKNOWN } } }
I20260925 15:04:23.577939 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:23.573399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.22.236.1:34715)
0925 15:04:23.573900 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:23.573904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:23.573920 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:04:23.573966 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:23.573969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:23.574025 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:23.574104 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:23.574469 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:23.575173 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:23.576488 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:23.576862 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:23.576953 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:23.577317 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:23.577439 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:23.577501 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:23.577626 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 15:04:23.577684 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":275,"threads_started":1}
I20260925 15:04:23.586380 25654 raft_consensus.cc:1261] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Refusing update from remote peer 1f9d90c6ae3548c5b66bef265e5b34ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:23.586410 25516 raft_consensus.cc:1261] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Refusing update from remote peer 1f9d90c6ae3548c5b66bef265e5b34ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:23.586891 25811 consensus_queue.cc:1035] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:23.587150 25794 consensus_queue.cc:1035] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:23.626417 25219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:23.626667 25219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:23.629513 25219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182: Sending DeleteTablet for 3 replicas of tablet 3314760a65dd4ae48fe902740f0cb857
I20260925 15:04:23.630266 25356 tablet_service.cc:1514] Processing DeleteTablet for tablet 3314760a65dd4ae48fe902740f0cb857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:23 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:38414
I20260925 15:04:23.630311 25635 tablet_service.cc:1514] Processing DeleteTablet for tablet 3314760a65dd4ae48fe902740f0cb857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:23 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:40006
I20260925 15:04:23.630409 25495 tablet_service.cc:1514] Processing DeleteTablet for tablet 3314760a65dd4ae48fe902740f0cb857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:23 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:50722
I20260925 15:04:23.630527 25816 tablet_replica.cc:330] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab: stopping tablet replica
I20260925 15:04:23.630546 25817 tablet_replica.cc:330] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585: stopping tablet replica
I20260925 15:04:23.630671 25818 tablet_replica.cc:330] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19: stopping tablet replica
I20260925 15:04:23.630707 25816 raft_consensus.cc:2229] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:04:23.630712 25817 raft_consensus.cc:2229] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:23.630848 25818 raft_consensus.cc:2229] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:23.630875 25817 raft_consensus.cc:2258] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:23.630976 25818 raft_consensus.cc:2258] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:23.630980 25816 raft_consensus.cc:2258] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:23.631882 25818 ts_tablet_manager.cc:1905] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:23.631884 25817 ts_tablet_manager.cc:1905] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:23.632066 25816 ts_tablet_manager.cc:1905] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:23.636315 25818 ts_tablet_manager.cc:1918] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:04:23.636439 25818 log.cc:1196] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/3314760a65dd4ae48fe902740f0cb857
I20260925 15:04:23.636938 25818 ts_tablet_manager.cc:1939] T 3314760a65dd4ae48fe902740f0cb857 P 4dc66b1554ef4cb2b5195429e74b8e19: Deleting consensus metadata
I20260925 15:04:23.637065 25219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:04:23.637827 25219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:04:23.638063 25205 catalog_manager.cc:4841] TS 4dc66b1554ef4cb2b5195429e74b8e19 (127.22.236.2:41355): tablet 3314760a65dd4ae48fe902740f0cb857 (table test-table [id=b2f33bf67f85457b8593d1080d7e7a77]) successfully deleted
I20260925 15:04:23.638244 25817 ts_tablet_manager.cc:1918] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:04:23.638244 25816 ts_tablet_manager.cc:1918] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:04:23.638561 25816 log.cc:1196] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/3314760a65dd4ae48fe902740f0cb857
I20260925 15:04:23.638978 25817 log.cc:1196] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/3314760a65dd4ae48fe902740f0cb857
I20260925 15:04:23.639068 25816 ts_tablet_manager.cc:1939] T 3314760a65dd4ae48fe902740f0cb857 P 1f9d90c6ae3548c5b66bef265e5b34ab: Deleting consensus metadata
I20260925 15:04:23.639396 25817 ts_tablet_manager.cc:1939] T 3314760a65dd4ae48fe902740f0cb857 P 0ab5582485d84191bde61fd1c9a18585: Deleting consensus metadata
I20260925 15:04:23.640128 25206 catalog_manager.cc:4841] TS 1f9d90c6ae3548c5b66bef265e5b34ab (127.22.236.1:34715): tablet 3314760a65dd4ae48fe902740f0cb857 (table test-table [id=b2f33bf67f85457b8593d1080d7e7a77]) successfully deleted
I20260925 15:04:23.640374 25206 catalog_manager.cc:4841] TS 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033): tablet 3314760a65dd4ae48fe902740f0cb857 (table test-table [id=b2f33bf67f85457b8593d1080d7e7a77]) successfully deleted
I20260925 15:04:23.643532 25356 tablet_service.cc:1467] Processing CreateTablet for tablet 223be85ceb9a4b7282448ddaa24bac87 (DEFAULT_TABLE table=test-table [id=87e6a1b119e94ce5899d027b49d19572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:23.643635 25495 tablet_service.cc:1467] Processing CreateTablet for tablet 223be85ceb9a4b7282448ddaa24bac87 (DEFAULT_TABLE table=test-table [id=87e6a1b119e94ce5899d027b49d19572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:23.643882 25356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223be85ceb9a4b7282448ddaa24bac87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:23.643817 25635 tablet_service.cc:1467] Processing CreateTablet for tablet 223be85ceb9a4b7282448ddaa24bac87 (DEFAULT_TABLE table=test-table [id=87e6a1b119e94ce5899d027b49d19572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:23.644074 25495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223be85ceb9a4b7282448ddaa24bac87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:23.644193 25635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223be85ceb9a4b7282448ddaa24bac87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:23.648034 25828 tablet_bootstrap.cc:492] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab: Bootstrap starting.
I20260925 15:04:23.649322 25828 tablet_bootstrap.cc:654] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:23.649952 25827 tablet_bootstrap.cc:492] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19: Bootstrap starting.
I20260925 15:04:23.651171 25827 tablet_bootstrap.cc:654] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:23.652717 25829 tablet_bootstrap.cc:492] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585: Bootstrap starting.
I20260925 15:04:23.652801 25827 tablet_bootstrap.cc:492] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19: No bootstrap required, opened a new log
I20260925 15:04:23.652947 25827 ts_tablet_manager.cc:1397] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 15:04:23.653571 25827 raft_consensus.cc:348] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.653748 25827 raft_consensus.cc:374] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:23.653827 25827 raft_consensus.cc:729] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc66b1554ef4cb2b5195429e74b8e19, State: Initialized, Role: FOLLOWER
I20260925 15:04:23.654045 25827 consensus_queue.cc:260] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.654311 25829 tablet_bootstrap.cc:654] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:23.654454 25827 ts_tablet_manager.cc:1428] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260925 15:04:23.656487 25829 tablet_bootstrap.cc:492] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585: No bootstrap required, opened a new log
I20260925 15:04:23.656670 25829 ts_tablet_manager.cc:1397] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 15:04:23.657406 25829 raft_consensus.cc:348] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.657653 25829 raft_consensus.cc:374] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:23.657724 25829 raft_consensus.cc:729] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab5582485d84191bde61fd1c9a18585, State: Initialized, Role: FOLLOWER
I20260925 15:04:23.657945 25829 consensus_queue.cc:260] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.658519 25829 ts_tablet_manager.cc:1428] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 15:04:23.661144 25828 tablet_bootstrap.cc:492] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab: No bootstrap required, opened a new log
I20260925 15:04:23.661332 25828 ts_tablet_manager.cc:1397] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260925 15:04:23.661970 25828 raft_consensus.cc:348] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.662200 25828 raft_consensus.cc:374] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:23.662305 25828 raft_consensus.cc:729] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9d90c6ae3548c5b66bef265e5b34ab, State: Initialized, Role: FOLLOWER
I20260925 15:04:23.662532 25828 consensus_queue.cc:260] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.662984 25828 ts_tablet_manager.cc:1428] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 15:04:23.974143 25789 raft_consensus.cc:482] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:04:23.974340 25789 raft_consensus.cc:504] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.975013 25789 leader_election.cc:288] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9d90c6ae3548c5b66bef265e5b34ab (127.22.236.1:34715), 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033)
I20260925 15:04:23.975587 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223be85ceb9a4b7282448ddaa24bac87" candidate_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" is_pre_election: true
I20260925 15:04:23.975603 25654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223be85ceb9a4b7282448ddaa24bac87" candidate_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab5582485d84191bde61fd1c9a18585" is_pre_election: true
I20260925 15:04:23.975857 25381 raft_consensus.cc:2454] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc66b1554ef4cb2b5195429e74b8e19 in term 0.
I20260925 15:04:23.975838 25654 raft_consensus.cc:2454] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dc66b1554ef4cb2b5195429e74b8e19 in term 0.
I20260925 15:04:23.976285 25449 leader_election.cc:302] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab5582485d84191bde61fd1c9a18585, 4dc66b1554ef4cb2b5195429e74b8e19; no voters:
I20260925 15:04:23.976537 25789 raft_consensus.cc:2781] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:04:23.976653 25789 raft_consensus.cc:482] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:04:23.976730 25789 raft_consensus.cc:3037] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:23.978571 25789 raft_consensus.cc:504] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.979023 25789 leader_election.cc:288] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [CANDIDATE]: Term 1 election: Requested vote from peers 1f9d90c6ae3548c5b66bef265e5b34ab (127.22.236.1:34715), 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033)
I20260925 15:04:23.979488 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223be85ceb9a4b7282448ddaa24bac87" candidate_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab"
I20260925 15:04:23.979524 25654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223be85ceb9a4b7282448ddaa24bac87" candidate_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab5582485d84191bde61fd1c9a18585"
I20260925 15:04:23.979714 25381 raft_consensus.cc:3037] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:23.979691 25654 raft_consensus.cc:3037] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:23.982529 25654 raft_consensus.cc:2454] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc66b1554ef4cb2b5195429e74b8e19 in term 1.
I20260925 15:04:23.982517 25381 raft_consensus.cc:2454] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dc66b1554ef4cb2b5195429e74b8e19 in term 1.
I20260925 15:04:23.982882 25449 leader_election.cc:302] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab5582485d84191bde61fd1c9a18585, 4dc66b1554ef4cb2b5195429e74b8e19; no voters:
I20260925 15:04:23.983130 25789 raft_consensus.cc:2781] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:23.983281 25789 raft_consensus.cc:686] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 LEADER]: Becoming Leader. State: Replica: 4dc66b1554ef4cb2b5195429e74b8e19, State: Running, Role: LEADER
I20260925 15:04:23.983544 25789 consensus_queue.cc:237] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } }
I20260925 15:04:23.990082 25220 catalog_manager.cc:5495] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dc66b1554ef4cb2b5195429e74b8e19 (127.22.236.2). New cstate: current_term: 1 leader_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dc66b1554ef4cb2b5195429e74b8e19" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 } health_report { overall_health: UNKNOWN } } }
I20260925 15:04:24.032063 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:24.027294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.22.236.2:41355)
0925 15:04:24.027818 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:24.027824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:24.027845 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:04:24.027898 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:24.027901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:24.027956 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:24.028028 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:24.028408 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:24.029185 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:24.030468 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:24.030873 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:24.030976 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:24.031384 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:24.031489 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:24.031558 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:24.031695 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:04:24.031763 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":295,"threads_started":1}
I20260925 15:04:24.039311 25381 raft_consensus.cc:1261] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Refusing update from remote peer 4dc66b1554ef4cb2b5195429e74b8e19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:24.039311 25654 raft_consensus.cc:1261] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Refusing update from remote peer 4dc66b1554ef4cb2b5195429e74b8e19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:24.039817 25802 consensus_queue.cc:1035] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9d90c6ae3548c5b66bef265e5b34ab" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:24.040104 25797 consensus_queue.cc:1035] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab5582485d84191bde61fd1c9a18585" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:04:24.058228 25795 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 15:04:25.629683 25220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:25.629894 25220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:25.633881 25220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e26e7b457a724a06aefec4b763906182: Sending DeleteTablet for 3 replicas of tablet 223be85ceb9a4b7282448ddaa24bac87
I20260925 15:04:25.634560 25356 tablet_service.cc:1514] Processing DeleteTablet for tablet 223be85ceb9a4b7282448ddaa24bac87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:25 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:38414
I20260925 15:04:25.634609 25496 tablet_service.cc:1514] Processing DeleteTablet for tablet 223be85ceb9a4b7282448ddaa24bac87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:25 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:50722
I20260925 15:04:25.634653 25635 tablet_service.cc:1514] Processing DeleteTablet for tablet 223be85ceb9a4b7282448ddaa24bac87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:25 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:40006
I20260925 15:04:25.635480 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 25291
I20260925 15:04:25.647365 25871 tablet_replica.cc:330] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab: stopping tablet replica
I20260925 15:04:25.647728 25871 raft_consensus.cc:2229] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:25.648048 25871 raft_consensus.cc:2258] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:25.649102 25871 ts_tablet_manager.cc:1905] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:25.650549 25870 tablet_replica.cc:330] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19: stopping tablet replica
I20260925 15:04:25.651031 25872 tablet_replica.cc:330] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585: stopping tablet replica
I20260925 15:04:25.651324 25870 raft_consensus.cc:2229] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:04:25.651594 25872 raft_consensus.cc:2229] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:25.651842 25872 raft_consensus.cc:2258] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:25.652343 25870 raft_consensus.cc:2258] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:25.653025 25872 ts_tablet_manager.cc:1905] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:25.655746 25870 ts_tablet_manager.cc:1905] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:25.655826 25871 ts_tablet_manager.cc:1918] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:04:25.656018 25871 log.cc:1196] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/223be85ceb9a4b7282448ddaa24bac87
I20260925 15:04:25.656548 25871 ts_tablet_manager.cc:1939] T 223be85ceb9a4b7282448ddaa24bac87 P 1f9d90c6ae3548c5b66bef265e5b34ab: Deleting consensus metadata
I20260925 15:04:25.657745 25206 catalog_manager.cc:4841] TS 1f9d90c6ae3548c5b66bef265e5b34ab (127.22.236.1:34715): tablet 223be85ceb9a4b7282448ddaa24bac87 (table test-table [id=87e6a1b119e94ce5899d027b49d19572]) successfully deleted
I20260925 15:04:25.659103 25872 ts_tablet_manager.cc:1918] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:04:25.659220 25872 log.cc:1196] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/223be85ceb9a4b7282448ddaa24bac87
I20260925 15:04:25.659570 25872 ts_tablet_manager.cc:1939] T 223be85ceb9a4b7282448ddaa24bac87 P 0ab5582485d84191bde61fd1c9a18585: Deleting consensus metadata
I20260925 15:04:25.660419 25206 catalog_manager.cc:4841] TS 0ab5582485d84191bde61fd1c9a18585 (127.22.236.3:41033): tablet 223be85ceb9a4b7282448ddaa24bac87 (table test-table [id=87e6a1b119e94ce5899d027b49d19572]) successfully deleted
Sep 25 15:04:25 dist-test-slave-nsrc krb5kdc[25175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348663, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:04:25.662160 25870 ts_tablet_manager.cc:1918] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:04:25.662290 25870 log.cc:1196] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/223be85ceb9a4b7282448ddaa24bac87
I20260925 15:04:25.662688 25870 ts_tablet_manager.cc:1939] T 223be85ceb9a4b7282448ddaa24bac87 P 4dc66b1554ef4cb2b5195429e74b8e19: Deleting consensus metadata
I20260925 15:04:25.663542 25205 catalog_manager.cc:4841] TS 4dc66b1554ef4cb2b5195429e74b8e19 (127.22.236.2:41355): tablet 223be85ceb9a4b7282448ddaa24bac87 (table test-table [id=87e6a1b119e94ce5899d027b49d19572]) successfully deleted
I20260925 15:04:25.668071 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:25.636684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.22.236.1:34715)
0925 15:04:25.651259 (+ 14575us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:25.651266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:25.651910 (+ 644us) server_negotiation.cc:407] Connection header received
0925 15:04:25.651971 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:25.651976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:25.652059 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:25.652172 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:25.653715 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:25.654565 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:25.656301 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:25.657041 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:25.657154 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:25.662778 (+ 5624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:25.662799 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:25.662820 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:25.662864 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:25.665516 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:25.666130 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:25.666138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:25.666140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:25.666282 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:25.666634 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:25.666638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:25.666640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:25.666958 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:25.667264 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:25.667549 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 15:04:25.667771 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14418,"thread_start_us":14305,"threads_started":1}
W20260925 15:04:25.668723 25422 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38444
W20260925 15:04:25.669364 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 25291: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:04:25.669548 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 25291
I20260925 15:04:25.687180 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 25431
Sep 25 15:04:25 dist-test-slave-nsrc krb5kdc[25175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348663, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:04:25.701090 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:25.687901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.22.236.2:41355)
0925 15:04:25.688363 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:25.688367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:25.688381 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:04:25.688433 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:25.688436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:25.688495 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:25.688579 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:25.689710 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:25.690381 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:25.691560 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:25.691966 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:25.692051 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:25.695974 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:25.695988 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:25.696003 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:25.696037 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:25.698727 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:25.699341 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:25.699349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:25.699351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:25.699469 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:25.699827 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:25.699831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:25.699833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:25.700239 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:25.700452 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:25.700670 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 15:04:25.700830 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
W20260925 15:04:25.701691 25561 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50770
W20260925 15:04:25.702258 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 25431: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:04:25.702380 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 25431
I20260925 15:04:25.721781 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 25570
Sep 25 15:04:25 dist-test-slave-nsrc krb5kdc[25175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348663, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:04:25.735646 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:25.722548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40010 (local address 127.22.236.3:41033)
0925 15:04:25.723063 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:25.723067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:25.723082 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:04:25.723134 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:25.723137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:25.723192 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:25.723278 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:25.724561 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:25.725288 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:25.726488 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:25.726875 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:25.726963 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:25.730822 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:25.730836 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:25.730851 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:25.730887 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:25.733397 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:25.733995 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:25.734004 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:25.734006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:25.734143 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:25.734438 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:25.734440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:25.734442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:25.734762 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:25.734962 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:25.735156 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 15:04:25.735318 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":253,"threads_started":1}
W20260925 15:04:25.736320 25700 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40010
W20260925 15:04:25.736927 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 25570: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:04:25.737076 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 25570
I20260925 15:04:25.750627 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 25188
I20260925 15:04:25.751758 25260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:25.862246 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 25188
2026-09-25T15:04:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8507 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:04:25 dist-test-slave-nsrc krb5kdc[25880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:04:25 dist-test-slave-nsrc krb5kdc[25880](info): set up 2 sockets
Sep 25 15:04:25 dist-test-slave-nsrc krb5kdc[25880](info): commencing operation
krb5kdc: starting...
W20260925 15:04:26.852308 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:04:26 dist-test-slave-nsrc krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:04:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:04:27.019025 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:41303
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:41303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:27.319847 25893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:27.320166 25893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:27.320293 25893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:27.327344 25893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:27.327461 25893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:27.327534 25893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:27.327606 25893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:27.327664 25893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:04:27.337204 25893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:41303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:41303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:27.338483 25893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:27.339701 25893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:27.351042 25898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:27.351089 25899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:27.351159 25901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:27.351706 25893 server_base.cc:1034] running on GCE node
I20260925 15:04:27.352523 25893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:27.353873 25893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:27.355070 25893 hybrid_clock.cc:648] HybridClock initialized: now 1790348667355028 us; error 28 us; skew 500 ppm
I20260925 15:04:27.355382 25893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:27 dist-test-slave-nsrc krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348667, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:27.358184 25893 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:04:27.359654 25893 webserver.cc:460] Webserver started at http://127.22.236.62:42547/ using document root <none> and password file <none>
I20260925 15:04:27.360342 25893 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:27.360453 25893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:27.360721 25893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:27.363543 25893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "4043d9f88f9648f080496fc15a80f936"
format_stamp: "Formatted at 2026-09-25 15:04:27 on dist-test-slave-nsrc"
server_key: "a9da69e0a888b302ea7496f87ecb88d6"
server_key_iv: "c5481755d0dd8982a0500b51914bedc4"
server_key_version: "encryptionkey@0"
I20260925 15:04:27.364315 25893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "4043d9f88f9648f080496fc15a80f936"
format_stamp: "Formatted at 2026-09-25 15:04:27 on dist-test-slave-nsrc"
server_key: "a9da69e0a888b302ea7496f87ecb88d6"
server_key_iv: "c5481755d0dd8982a0500b51914bedc4"
server_key_version: "encryptionkey@0"
I20260925 15:04:27.369208 25893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 15:04:27.372720 25908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:27.374135 25893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 15:04:27.374297 25893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "4043d9f88f9648f080496fc15a80f936"
format_stamp: "Formatted at 2026-09-25 15:04:27 on dist-test-slave-nsrc"
server_key: "a9da69e0a888b302ea7496f87ecb88d6"
server_key_iv: "c5481755d0dd8982a0500b51914bedc4"
server_key_version: "encryptionkey@0"
I20260925 15:04:27.374485 25893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:27.388598 25893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:27.392521 25893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:27.392746 25893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:27.409242 25893 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:41303
I20260925 15:04:27.409253 25959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:41303 every 8 connection(s)
I20260925 15:04:27.410539 25893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:27.413203 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 25893
I20260925 15:04:27.413390 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:27.413992 23472 external_mini_cluster.cc:1428] Setting key 83f043ca82a29928c05ebcd254e1a2fc
I20260925 15:04:27.414410 25960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:27.427424 25960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936: Bootstrap starting.
Sep 25 15:04:27 dist-test-slave-nsrc krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:27.430888 25960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:27.432178 25960 log.cc:824] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:27.435431 25960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936: No bootstrap required, opened a new log
I20260925 15:04:27.437180 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:27.415755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.22.236.62:41303)
0925 15:04:27.416576 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:27.416591 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:27.416637 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:04:27.418771 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:27.418797 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:27.419269 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:27.419686 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:27.420945 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:27.422857 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:27.424126 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:27.425185 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:27.425376 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:27.428983 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:27.428994 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:27.429013 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:27.429065 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:27.432651 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:27.433554 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:27.433562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:27.433571 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:27.433803 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:27.434155 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:27.434160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:27.434163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:27.434836 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:27.435093 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:27.435435 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 15:04:27.435863 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":342,"threads_started":1}
I20260925 15:04:27.443382 25960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4043d9f88f9648f080496fc15a80f936" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 41303 } }
I20260925 15:04:27.443686 25960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:27.443758 25960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4043d9f88f9648f080496fc15a80f936, State: Initialized, Role: FOLLOWER
I20260925 15:04:27.444255 25960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4043d9f88f9648f080496fc15a80f936" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 41303 } }
I20260925 15:04:27.444451 25960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:27.444563 25960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:27.444746 25960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:27.446817 25960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4043d9f88f9648f080496fc15a80f936" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 41303 } }
I20260925 15:04:27.447280 25960 leader_election.cc:302] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4043d9f88f9648f080496fc15a80f936; no voters:
I20260925 15:04:27.447800 25960 leader_election.cc:288] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:27.447924 25965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:27.448773 25965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [term 1 LEADER]: Becoming Leader. State: Replica: 4043d9f88f9648f080496fc15a80f936, State: Running, Role: LEADER
I20260925 15:04:27.449290 25965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4043d9f88f9648f080496fc15a80f936" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 41303 } }
I20260925 15:04:27.449805 25960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:27.452168 25967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043d9f88f9648f080496fc15a80f936. Latest consensus state: current_term: 1 leader_uuid: "4043d9f88f9648f080496fc15a80f936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4043d9f88f9648f080496fc15a80f936" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 41303 } } }
I20260925 15:04:27.452172 25966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4043d9f88f9648f080496fc15a80f936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4043d9f88f9648f080496fc15a80f936" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 41303 } } }
I20260925 15:04:27.452586 25967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:27.452586 25966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:27.454003 25973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:27.458989 25973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:27.467038 25973 catalog_manager.cc:1347] Generated new cluster ID: b2d09c64092d44b0a2680b7b980323bb
I20260925 15:04:27.467142 25973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:27.489393 25973 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:27.490696 25973 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:27.505688 25973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4043d9f88f9648f080496fc15a80f936: Generated new TSK 0
I20260925 15:04:27.506512 25973 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:04:27.563771 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:41303
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:27.876957 25986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:27.877220 25986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:27.877318 25986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:27.884419 25986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:27.884541 25986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:27.884690 25986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:04:27.894327 25986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:41303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:27.895548 25986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:27.896874 25986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:27.909065 25991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:27.909032 25992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:27.909121 25994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:27.909608 25986 server_base.cc:1034] running on GCE node
I20260925 15:04:27.910228 25986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:27.911068 25986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:27.912257 25986 hybrid_clock.cc:648] HybridClock initialized: now 1790348667912197 us; error 40 us; skew 500 ppm
I20260925 15:04:27.912552 25986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:27 dist-test-slave-nsrc krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348667, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:27.915383 25986 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:04:27.916642 25986 webserver.cc:460] Webserver started at http://127.22.236.1:33167/ using document root <none> and password file <none>
I20260925 15:04:27.917246 25986 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:27.917336 25986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:27.917567 25986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:27.920199 25986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "6d4de3fad61a47079efc4c3d7a9d9cbc"
format_stamp: "Formatted at 2026-09-25 15:04:27 on dist-test-slave-nsrc"
server_key: "5b760012c3698bc65b0a851fd4f16d19"
server_key_iv: "bad5c8669d8b7d7ff04f11c1a770b024"
server_key_version: "encryptionkey@0"
I20260925 15:04:27.920879 25986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "6d4de3fad61a47079efc4c3d7a9d9cbc"
format_stamp: "Formatted at 2026-09-25 15:04:27 on dist-test-slave-nsrc"
server_key: "5b760012c3698bc65b0a851fd4f16d19"
server_key_iv: "bad5c8669d8b7d7ff04f11c1a770b024"
server_key_version: "encryptionkey@0"
I20260925 15:04:27.925441 25986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 15:04:27.928442 26001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:27.929589 25986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:04:27.929721 25986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "6d4de3fad61a47079efc4c3d7a9d9cbc"
format_stamp: "Formatted at 2026-09-25 15:04:27 on dist-test-slave-nsrc"
server_key: "5b760012c3698bc65b0a851fd4f16d19"
server_key_iv: "bad5c8669d8b7d7ff04f11c1a770b024"
server_key_version: "encryptionkey@0"
I20260925 15:04:27.929885 25986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:27.944979 25986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:27.948792 25986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:27.949030 25986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:27.949831 25986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:27.951193 25986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:27.951261 25986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:27.951387 25986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:27.951431 25986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:27.979931 25986 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:39267
I20260925 15:04:27.979985 26113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:39267 every 8 connection(s)
I20260925 15:04:27.981120 25986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:04:27.991518 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 25986
I20260925 15:04:27.991743 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:04:27.992530 23472 external_mini_cluster.cc:1428] Setting key 715c2a38e943a1ec7120af35fedb4733
Sep 25 15:04:27 dist-test-slave-nsrc krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348667, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:27.999428 26116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:27.983304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:56733 (local address 127.22.236.62:41303)
0925 15:04:27.983696 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:27.983700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:27.984656 (+ 956us) server_negotiation.cc:407] Connection header received
0925 15:04:27.986356 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:27.986362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:27.986435 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:27.986528 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:27.987935 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:27.988687 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:27.990152 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:27.990572 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:27.990678 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:27.994327 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:27.994334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:27.994356 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:27.994393 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:27.996729 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:27.997382 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:27.997387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:27.997391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:27.997501 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:27.997882 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:27.997887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:27.997890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:27.998139 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:27.998269 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:27.998880 (+ 611us) server_negotiation.cc:299] Negotiation successful
0925 15:04:27.999082 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":201,"threads_started":1}
I20260925 15:04:28.000133 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:27.983838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:41303 (local address 127.22.236.1:56733)
0925 15:04:27.984513 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:27.984555 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:27.986103 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:27.986751 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:27.986759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:27.987249 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:27.987775 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:27.987791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:27.988824 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:27.988828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:27.990024 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:27.990035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:27.990778 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:27.990785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:27.991006 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:27.991639 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:27.991672 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:27.994080 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:27.996860 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:27.996869 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:27.996885 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:27.997268 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:27.997598 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:27.997603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:27.997608 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:27.997781 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:27.998272 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:27.998278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:27.998567 (+ 289us) client_negotiation.cc:769] Sending connection context
0925 15:04:27.998784 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 15:04:27.999138 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":272,"threads_started":1}
I20260925 15:04:28.001530 26114 heartbeater.cc:347] Connected to a master server at 127.22.236.62:41303
I20260925 15:04:28.001785 26114 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:28.002434 26114 heartbeater.cc:511] Master 127.22.236.62:41303 requested a full tablet report, sending...
I20260925 15:04:28.004567 25925 ts_manager.cc:194] Registered new tserver with Master: 6d4de3fad61a47079efc4c3d7a9d9cbc (127.22.236.1:39267)
I20260925 15:04:28.006634 25925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:56733
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:04:28.028484 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:41303
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:28.323289 26120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:28.323576 26120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:28.323697 26120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:28.330687 26120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:28.330796 26120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:28.330945 26120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:04:28.340291 26120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:41303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:28.341467 26120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:28.342679 26120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:28.354707 26126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:28.354825 26125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:28.354734 26128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:28.355319 26120 server_base.cc:1034] running on GCE node
I20260925 15:04:28.355821 26120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:28.356658 26120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:28.357843 26120 hybrid_clock.cc:648] HybridClock initialized: now 1790348668357804 us; error 30 us; skew 500 ppm
I20260925 15:04:28.358103 26120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:28 dist-test-slave-nsrc krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348668, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:28.368459 26120 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:04:28.369791 26120 webserver.cc:460] Webserver started at http://127.22.236.2:36473/ using document root <none> and password file <none>
I20260925 15:04:28.370491 26120 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:28.370589 26120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:28.370834 26120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:28.373642 26120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "8a61caef768e4907821364824e8f8d41"
format_stamp: "Formatted at 2026-09-25 15:04:28 on dist-test-slave-nsrc"
server_key: "58a5a838d8a3ca5d01f84abe5d662a6c"
server_key_iv: "e7639185f77891f01acc6a7e1201329c"
server_key_version: "encryptionkey@0"
I20260925 15:04:28.374356 26120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "8a61caef768e4907821364824e8f8d41"
format_stamp: "Formatted at 2026-09-25 15:04:28 on dist-test-slave-nsrc"
server_key: "58a5a838d8a3ca5d01f84abe5d662a6c"
server_key_iv: "e7639185f77891f01acc6a7e1201329c"
server_key_version: "encryptionkey@0"
I20260925 15:04:28.379438 26120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 15:04:28.382725 26135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:28.383952 26120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:04:28.384102 26120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "8a61caef768e4907821364824e8f8d41"
format_stamp: "Formatted at 2026-09-25 15:04:28 on dist-test-slave-nsrc"
server_key: "58a5a838d8a3ca5d01f84abe5d662a6c"
server_key_iv: "e7639185f77891f01acc6a7e1201329c"
server_key_version: "encryptionkey@0"
I20260925 15:04:28.384313 26120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:28.402390 26120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:28.406234 26120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:28.406490 26120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:28.407378 26120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:28.408700 26120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:28.408789 26120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:28.408901 26120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:28.408965 26120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:28.437815 26120 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:38185
I20260925 15:04:28.437855 26247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:38185 every 8 connection(s)
I20260925 15:04:28.438944 26120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:04:28.441459 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 26120
I20260925 15:04:28.441638 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:04:28.442174 23472 external_mini_cluster.cc:1428] Setting key 728f8212f289e0772bd26094774c0046
Sep 25 15:04:28 dist-test-slave-nsrc krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348668, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
I20260925 15:04:28.459275 26116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:28.441194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:33741 (local address 127.22.236.62:41303)
0925 15:04:28.441350 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:28.441354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:28.442673 (+ 1319us) server_negotiation.cc:407] Connection header received
0925 15:04:28.444709 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:28.444712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:28.444785 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:28.444862 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:28.446685 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:28.447373 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:28.449414 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:28.449852 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:28.449959 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:28.453523 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:28.453529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:28.453534 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:28.453571 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:28.456461 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:28.457116 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:28.457121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:28.457125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:28.457272 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:28.457659 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:28.457665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:28.457668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:28.457984 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:28.458133 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:28.458747 (+ 614us) server_negotiation.cc:299] Negotiation successful
0925 15:04:28.458960 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:04:28.460063 26250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:28.441812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:41303 (local address 127.22.236.2:33741)
0925 15:04:28.442531 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:28.442568 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:28.444443 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:28.445102 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:28.445113 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:28.445816 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:28.446544 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:28.446566 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:28.447503 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:28.447507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:28.449281 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:28.449292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:28.450058 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:28.450066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:28.450283 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:28.450948 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:28.450973 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:28.453300 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:28.456586 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:28.456593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:28.456604 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:28.456992 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:28.457381 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:28.457384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:28.457386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:28.457552 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:28.458084 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:28.458089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:28.458382 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 15:04:28.458613 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 15:04:28.459000 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":270,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
I20260925 15:04:28.461418 26248 heartbeater.cc:347] Connected to a master server at 127.22.236.62:41303
I20260925 15:04:28.461686 26248 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:28.462378 26248 heartbeater.cc:511] Master 127.22.236.62:41303 requested a full tablet report, sending...
I20260925 15:04:28.463665 25925 ts_manager.cc:194] Registered new tserver with Master: 8a61caef768e4907821364824e8f8d41 (127.22.236.2:38185)
I20260925 15:04:28.464457 25925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:33741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:04:28.481092 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:41303
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:28.780933 26253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:28.781206 26253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:28.781343 26253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:28.788355 26253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:28.788465 26253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:28.788617 26253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:04:28.797899 26253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:41303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:28.799134 26253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:28.800382 26253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:28.812382 26258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:28.812461 26259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:28.812424 26261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:28.813293 26253 server_base.cc:1034] running on GCE node
I20260925 15:04:28.813930 26253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:28.814769 26253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:28.815953 26253 hybrid_clock.cc:648] HybridClock initialized: now 1790348668815905 us; error 46 us; skew 500 ppm
I20260925 15:04:28.816263 26253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:28 dist-test-slave-nsrc krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348668, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:28.818995 26253 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:04:28.820513 26253 webserver.cc:460] Webserver started at http://127.22.236.3:40287/ using document root <none> and password file <none>
I20260925 15:04:28.821166 26253 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:28.821275 26253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:28.821549 26253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:28.824308 26253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "299f56837bb64594a5dc2ab904e2fac4"
format_stamp: "Formatted at 2026-09-25 15:04:28 on dist-test-slave-nsrc"
server_key: "af3706b76e37f427459704882411d706"
server_key_iv: "21b684a2ab7e63a29ab757cac0fd1531"
server_key_version: "encryptionkey@0"
I20260925 15:04:28.825033 26253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "299f56837bb64594a5dc2ab904e2fac4"
format_stamp: "Formatted at 2026-09-25 15:04:28 on dist-test-slave-nsrc"
server_key: "af3706b76e37f427459704882411d706"
server_key_iv: "21b684a2ab7e63a29ab757cac0fd1531"
server_key_version: "encryptionkey@0"
I20260925 15:04:28.829633 26253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:04:28.832798 26268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:28.834157 26253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:28.834293 26253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "299f56837bb64594a5dc2ab904e2fac4"
format_stamp: "Formatted at 2026-09-25 15:04:28 on dist-test-slave-nsrc"
server_key: "af3706b76e37f427459704882411d706"
server_key_iv: "21b684a2ab7e63a29ab757cac0fd1531"
server_key_version: "encryptionkey@0"
I20260925 15:04:28.834460 26253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:28.847949 26253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:28.851775 26253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:28.851996 26253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:28.852826 26253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:28.854169 26253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:28.854242 26253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:28.854333 26253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:28.854378 26253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:28.880640 26253 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:46175
I20260925 15:04:28.880673 26380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:46175 every 8 connection(s)
I20260925 15:04:28.881871 26253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:04:28.885007 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 26253
I20260925 15:04:28.885191 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:04:28.885701 23472 external_mini_cluster.cc:1428] Setting key 851d2c9d441dde0d6fbd2ea20e3bfd2c
Sep 25 15:04:28 dist-test-slave-nsrc krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348668, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:28.900116 26116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:28.884160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:33609 (local address 127.22.236.62:41303)
0925 15:04:28.884336 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:28.884342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:28.885685 (+ 1343us) server_negotiation.cc:407] Connection header received
0925 15:04:28.887860 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:28.887865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:28.887943 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:28.888045 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:28.889558 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:28.890269 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:28.891685 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:28.892088 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:28.892213 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:28.895130 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:28.895134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:28.895138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:28.895167 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:28.897525 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:28.898118 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:28.898122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:28.898124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:28.898237 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:28.898566 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:28.898569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:28.898570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:28.898830 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:28.898943 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:28.899644 (+ 701us) server_negotiation.cc:299] Negotiation successful
0925 15:04:28.899802 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:04:28.901187 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:28.884710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:41303 (local address 127.22.236.3:33609)
0925 15:04:28.885525 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:28.885576 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:28.887614 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:28.888267 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:28.888278 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:28.888890 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:28.889415 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:28.889431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:28.890383 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:28.890386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:28.891564 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:28.891575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:28.892269 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:28.892274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:28.892477 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:28.893008 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:28.893028 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:28.894936 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:28.897629 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:28.897635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:28.897646 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:28.898002 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:28.898319 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:28.898322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:28.898324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:28.898492 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:28.898951 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:28.898958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:28.899300 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 15:04:28.899557 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 15:04:28.900008 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":291,"threads_started":1}
I20260925 15:04:28.902266 26381 heartbeater.cc:347] Connected to a master server at 127.22.236.62:41303
I20260925 15:04:28.902535 26381 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:28.903167 26381 heartbeater.cc:511] Master 127.22.236.62:41303 requested a full tablet report, sending...
I20260925 15:04:28.904405 25925 ts_manager.cc:194] Registered new tserver with Master: 299f56837bb64594a5dc2ab904e2fac4 (127.22.236.3:46175)
I20260925 15:04:28.905189 25925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:33609
I20260925 15:04:28.912431 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:04:28.937457 26116 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:28.935302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.22.236.62:41303)
0925 15:04:28.935475 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:28.935481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:28.935877 (+ 396us) server_negotiation.cc:407] Connection header received
0925 15:04:28.936026 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:28.936030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:28.936093 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:28.936226 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:28.937251 (+ 1025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40422: BlockingRecv error: recv got EOF from 127.0.0.1:40422 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:04:28.940667 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 25986
W20260925 15:04:28.946350 26392 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:28.943102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.22.236.1:39267)
0925 15:04:28.943687 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:28.943698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:28.944598 (+ 900us) server_negotiation.cc:407] Connection header received
0925 15:04:28.944874 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:28.944884 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:28.945152 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:28.945315 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:28.946016 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57914: BlockingRecv error: recv got EOF from 127.0.0.1:57914 (error 108)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":285,"threads_started":1}
W20260925 15:04:28.946646 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 25986: Not authorized: Client connection negotiation failed: client connection to 127.22.236.1:39267: server requires authentication, but client does not have Kerberos credentials available
I20260925 15:04:28.946805 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 25986
I20260925 15:04:28.959004 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 26120
W20260925 15:04:28.961545 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 26120: Not authorized: Client connection negotiation failed: client connection to 127.22.236.2:38185: server requires authentication, but client does not have Kerberos credentials available
I20260925 15:04:28.961688 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 26120
I20260925 15:04:28.972851 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 26253
W20260925 15:04:28.975235 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 26253: Not authorized: Client connection negotiation failed: client connection to 127.22.236.3:46175: server requires authentication, but client does not have Kerberos credentials available
I20260925 15:04:28.975360 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 26253
I20260925 15:04:28.986594 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 25893
I20260925 15:04:28.987643 25955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:29.086817 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 25893
2026-09-25T15:04:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3224 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:04:29 dist-test-slave-nsrc krb5kdc[26398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:04:29 dist-test-slave-nsrc krb5kdc[26398](info): set up 2 sockets
Sep 25 15:04:29 dist-test-slave-nsrc krb5kdc[26398](info): commencing operation
krb5kdc: starting...
W20260925 15:04:30.092445 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:04:30 dist-test-slave-nsrc krb5kdc[26398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:04:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:04:30.273931 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:34841
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:36621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:34841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:30.575963 26411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:30.576267 26411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:30.576404 26411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:30.583472 26411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:30.583585 26411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:30.583662 26411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:30.583737 26411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:30.583801 26411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:04:30.593353 26411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:36621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:34841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:34841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:30.594583 26411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:30.595831 26411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:30.606382 26419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:30.606470 26417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:30.606638 26416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:30.607374 26411 server_base.cc:1034] running on GCE node
I20260925 15:04:30.608031 26411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:30.609287 26411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:30.610482 26411 hybrid_clock.cc:648] HybridClock initialized: now 1790348670610448 us; error 26 us; skew 500 ppm
I20260925 15:04:30.610776 26411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:30 dist-test-slave-nsrc krb5kdc[26398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348670, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:30.613584 26411 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:04:30.614830 26411 webserver.cc:460] Webserver started at http://127.22.236.62:36083/ using document root <none> and password file <none>
I20260925 15:04:30.615419 26411 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:30.615511 26411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:30.615742 26411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:30.618484 26411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "fc36a12b17e54ad7b97373a3e96adb7b"
format_stamp: "Formatted at 2026-09-25 15:04:30 on dist-test-slave-nsrc"
server_key: "f326803c84c66b2278aba294b543b168"
server_key_iv: "0022841a34fb56e22488e72e4be4b571"
server_key_version: "encryptionkey@0"
I20260925 15:04:30.619158 26411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "fc36a12b17e54ad7b97373a3e96adb7b"
format_stamp: "Formatted at 2026-09-25 15:04:30 on dist-test-slave-nsrc"
server_key: "f326803c84c66b2278aba294b543b168"
server_key_iv: "0022841a34fb56e22488e72e4be4b571"
server_key_version: "encryptionkey@0"
I20260925 15:04:30.623737 26411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 15:04:30.626870 26426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:30.628116 26411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:04:30.628324 26411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "fc36a12b17e54ad7b97373a3e96adb7b"
format_stamp: "Formatted at 2026-09-25 15:04:30 on dist-test-slave-nsrc"
server_key: "f326803c84c66b2278aba294b543b168"
server_key_iv: "0022841a34fb56e22488e72e4be4b571"
server_key_version: "encryptionkey@0"
I20260925 15:04:30.628494 26411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:30.648447 26411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:30.651978 26411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:30.652242 26411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:30.667987 26411 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:34841
I20260925 15:04:30.667989 26477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:34841 every 8 connection(s)
I20260925 15:04:30.669270 26411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:30.672792 26478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:30.678380 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 26411
I20260925 15:04:30.678565 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:30.679163 23472 external_mini_cluster.cc:1428] Setting key d90caa16aeec4108528188be9f699b42
I20260925 15:04:30.683418 26478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b: Bootstrap starting.
I20260925 15:04:30.687069 26478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:30.688295 26478 log.cc:824] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:30.691220 26478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b: No bootstrap required, opened a new log
Sep 25 15:04:30 dist-test-slave-nsrc krb5kdc[26398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:30.700935 26478 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc36a12b17e54ad7b97373a3e96adb7b" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 34841 } }
I20260925 15:04:30.701376 26478 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:30.701483 26478 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc36a12b17e54ad7b97373a3e96adb7b, State: Initialized, Role: FOLLOWER
I20260925 15:04:30.702119 26478 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc36a12b17e54ad7b97373a3e96adb7b" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 34841 } }
I20260925 15:04:30.702391 26478 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:30.702523 26478 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:30.702682 26478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:30.703110 26481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:30.680996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.22.236.62:34841)
0925 15:04:30.681738 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:30.681753 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:30.681801 (+ 48us) server_negotiation.cc:407] Connection header received
0925 15:04:30.684078 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:30.684106 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:30.684605 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:30.685085 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:30.686313 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:30.688207 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:30.689748 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:30.690711 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:30.690894 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:30.695086 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:30.695097 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:30.695116 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:30.695171 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:30.698912 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:30.699525 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:30.699534 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:30.699543 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:30.699746 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:30.700038 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:30.700044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:30.700047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:30.700756 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:30.701023 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:30.701383 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 15:04:30.701796 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":286,"threads_started":1}
I20260925 15:04:30.705520 26478 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc36a12b17e54ad7b97373a3e96adb7b" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 34841 } }
I20260925 15:04:30.706048 26478 leader_election.cc:302] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc36a12b17e54ad7b97373a3e96adb7b; no voters:
I20260925 15:04:30.706632 26478 leader_election.cc:288] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:30.706717 26483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:30.706842 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:04:30.708386 26483 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [term 1 LEADER]: Becoming Leader. State: Replica: fc36a12b17e54ad7b97373a3e96adb7b, State: Running, Role: LEADER
I20260925 15:04:30.708798 26478 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:30.708963 26483 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc36a12b17e54ad7b97373a3e96adb7b" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 34841 } }
I20260925 15:04:30.711992 26485 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc36a12b17e54ad7b97373a3e96adb7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc36a12b17e54ad7b97373a3e96adb7b" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 34841 } } }
I20260925 15:04:30.712412 26485 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:30.711972 26484 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc36a12b17e54ad7b97373a3e96adb7b. Latest consensus state: current_term: 1 leader_uuid: "fc36a12b17e54ad7b97373a3e96adb7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc36a12b17e54ad7b97373a3e96adb7b" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 34841 } } }
I20260925 15:04:30.712992 26484 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:30.724360 26491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:30.730967 26491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:30.740187 26491 catalog_manager.cc:1347] Generated new cluster ID: 65bc6b0f1327423082b655a76b9d6b6b
I20260925 15:04:30.740288 26491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:30.752950 26481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:30.746090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51168 (local address 127.22.236.62:34841)
0925 15:04:30.746258 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:30.746264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:30.746773 (+ 509us) server_negotiation.cc:407] Connection header received
0925 15:04:30.746942 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:30.746948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:30.747016 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:30.747095 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:30.748058 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:30.748858 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:30.750066 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:30.750492 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:30.750593 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:30.751566 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:30.751571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:30.751574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:30.752194 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:30.752263 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:30.752290 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:30.752384 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:30.752507 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 15:04:30.752598 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:04:30.797524 26491 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:30.798779 26491 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:30.807636 26491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc36a12b17e54ad7b97373a3e96adb7b: Generated new TSK 0
I20260925 15:04:30.808415 26491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:04:30.817762 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 26411
I20260925 15:04:30.818882 26473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:30.908281 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 26411
2026-09-25T15:04:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1815 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:04:30 dist-test-slave-nsrc krb5kdc[26512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:04:30 dist-test-slave-nsrc krb5kdc[26512](info): set up 2 sockets
Sep 25 15:04:30 dist-test-slave-nsrc krb5kdc[26512](info): commencing operation
krb5kdc: starting...
W20260925 15:04:31.888319 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:04:31 dist-test-slave-nsrc krb5kdc[26512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:04:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:04:32.064101 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:36057
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:35929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:36057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:32.363063 26525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:32.363315 26525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:32.363406 26525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:32.370309 26525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:32.370407 26525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:32.370458 26525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:32.370510 26525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:32.370551 26525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:04:32.380106 26525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:35929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:36057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:36057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:32.381383 26525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:32.382572 26525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:32.393468 26530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:32.393590 26531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:32.393589 26533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:32.394572 26525 server_base.cc:1034] running on GCE node
I20260925 15:04:32.395383 26525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:32.396778 26525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:32.397966 26525 hybrid_clock.cc:648] HybridClock initialized: now 1790348672397918 us; error 34 us; skew 500 ppm
I20260925 15:04:32.398267 26525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:32 dist-test-slave-nsrc krb5kdc[26512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348672, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:32.401152 26525 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:04:32.402485 26525 webserver.cc:460] Webserver started at http://127.22.236.62:33073/ using document root <none> and password file <none>
I20260925 15:04:32.403100 26525 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:32.403201 26525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:32.403468 26525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:32.406239 26525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "d3ad078cff7d4b75a6e5c79b010bc4b9"
format_stamp: "Formatted at 2026-09-25 15:04:32 on dist-test-slave-nsrc"
server_key: "f4ce7561abe471ab10ab3b0fa99c57be"
server_key_iv: "ac2663e242f9eeaeee99f3b687d7c00a"
server_key_version: "encryptionkey@0"
I20260925 15:04:32.406935 26525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "d3ad078cff7d4b75a6e5c79b010bc4b9"
format_stamp: "Formatted at 2026-09-25 15:04:32 on dist-test-slave-nsrc"
server_key: "f4ce7561abe471ab10ab3b0fa99c57be"
server_key_iv: "ac2663e242f9eeaeee99f3b687d7c00a"
server_key_version: "encryptionkey@0"
I20260925 15:04:32.411588 26525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 15:04:32.414796 26540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:32.416056 26525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:04:32.416234 26525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "d3ad078cff7d4b75a6e5c79b010bc4b9"
format_stamp: "Formatted at 2026-09-25 15:04:32 on dist-test-slave-nsrc"
server_key: "f4ce7561abe471ab10ab3b0fa99c57be"
server_key_iv: "ac2663e242f9eeaeee99f3b687d7c00a"
server_key_version: "encryptionkey@0"
I20260925 15:04:32.416397 26525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:32.455494 26525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:32.459298 26525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:32.459548 26525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:32.475713 26525 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:36057
I20260925 15:04:32.475742 26591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:36057 every 8 connection(s)
I20260925 15:04:32.477035 26525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:32.480566 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 26525
I20260925 15:04:32.480685 26592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:32.480744 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:32.481335 23472 external_mini_cluster.cc:1428] Setting key dee45f4b81ce5b813a81112583b67d94
I20260925 15:04:32.493167 26592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9: Bootstrap starting.
Sep 25 15:04:32 dist-test-slave-nsrc krb5kdc[26512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:32.496125 26592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:32.497117 26592 log.cc:824] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:32.499725 26592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9: No bootstrap required, opened a new log
I20260925 15:04:32.503252 26595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:32.483172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.22.236.62:36057)
0925 15:04:32.483924 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:32.483938 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:32.483985 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:04:32.487220 (+ 3235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:32.487249 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:32.487714 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:32.488081 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:32.489473 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:32.490758 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:32.491975 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:32.492682 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:32.492847 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:32.496719 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:32.496727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:32.496741 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:32.496776 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:32.499520 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:32.500181 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:32.500188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:32.500194 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:32.500356 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:32.500695 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:32.500702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:32.500704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:32.501197 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:32.501485 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:32.501890 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 15:04:32.502302 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":306,"threads_started":1}
I20260925 15:04:32.507720 26592 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ad078cff7d4b75a6e5c79b010bc4b9" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36057 } }
I20260925 15:04:32.508183 26592 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:32.508311 26592 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ad078cff7d4b75a6e5c79b010bc4b9, State: Initialized, Role: FOLLOWER
I20260925 15:04:32.508945 26592 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ad078cff7d4b75a6e5c79b010bc4b9" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36057 } }
I20260925 15:04:32.509164 26592 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:32.509291 26592 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:32.509434 26592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:32.511428 26592 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ad078cff7d4b75a6e5c79b010bc4b9" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36057 } }
I20260925 15:04:32.511873 26592 leader_election.cc:302] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3ad078cff7d4b75a6e5c79b010bc4b9; no voters:
I20260925 15:04:32.512475 26592 leader_election.cc:288] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:32.512598 26597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:32.513206 26597 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [term 1 LEADER]: Becoming Leader. State: Replica: d3ad078cff7d4b75a6e5c79b010bc4b9, State: Running, Role: LEADER
I20260925 15:04:32.513695 26597 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ad078cff7d4b75a6e5c79b010bc4b9" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36057 } }
I20260925 15:04:32.514411 26592 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:32.516234 26599 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3ad078cff7d4b75a6e5c79b010bc4b9. Latest consensus state: current_term: 1 leader_uuid: "d3ad078cff7d4b75a6e5c79b010bc4b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ad078cff7d4b75a6e5c79b010bc4b9" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36057 } } }
I20260925 15:04:32.516575 26599 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:32.517035 26598 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3ad078cff7d4b75a6e5c79b010bc4b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ad078cff7d4b75a6e5c79b010bc4b9" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36057 } } }
I20260925 15:04:32.517359 26598 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:32.525018 26603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:32.531554 26603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:32.540253 26603 catalog_manager.cc:1347] Generated new cluster ID: 3f995a42c7534bf9af39558cbc35ae5a
I20260925 15:04:32.540369 26603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:32.566776 26603 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:32.568024 26603 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:32.573470 26603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3ad078cff7d4b75a6e5c79b010bc4b9: Generated new TSK 0
I20260925 15:04:32.574221 26603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:04:32.629603 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36057
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:35929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:32.954857 26618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:32.955134 26618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:32.955255 26618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:32.962486 26618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:32.962605 26618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:32.962769 26618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:04:32.972332 26618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:35929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:32.973589 26618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:32.974853 26618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:32.987181 26623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:32.987272 26624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:32.987221 26626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:32.988322 26618 server_base.cc:1034] running on GCE node
I20260925 15:04:32.988855 26618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:32.989794 26618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:32.990988 26618 hybrid_clock.cc:648] HybridClock initialized: now 1790348672990938 us; error 41 us; skew 500 ppm
I20260925 15:04:32.991247 26618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:32 dist-test-slave-nsrc krb5kdc[26512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348672, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:32.994128 26618 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:04:32.995426 26618 webserver.cc:460] Webserver started at http://127.22.236.1:36161/ using document root <none> and password file <none>
I20260925 15:04:32.996088 26618 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:32.996222 26618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:32.996500 26618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:32.999351 26618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "3f92a15875a54a81b66e7e2f8e2182e2"
format_stamp: "Formatted at 2026-09-25 15:04:32 on dist-test-slave-nsrc"
server_key: "04f436fd9b4a5e097911ca676186eb03"
server_key_iv: "fa97448d9e80ff8b9e6d20485f54e893"
server_key_version: "encryptionkey@0"
I20260925 15:04:33.000041 26618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "3f92a15875a54a81b66e7e2f8e2182e2"
format_stamp: "Formatted at 2026-09-25 15:04:32 on dist-test-slave-nsrc"
server_key: "04f436fd9b4a5e097911ca676186eb03"
server_key_iv: "fa97448d9e80ff8b9e6d20485f54e893"
server_key_version: "encryptionkey@0"
I20260925 15:04:33.004963 26618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 15:04:33.008095 26633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:33.009410 26618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:04:33.009557 26618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "3f92a15875a54a81b66e7e2f8e2182e2"
format_stamp: "Formatted at 2026-09-25 15:04:32 on dist-test-slave-nsrc"
server_key: "04f436fd9b4a5e097911ca676186eb03"
server_key_iv: "fa97448d9e80ff8b9e6d20485f54e893"
server_key_version: "encryptionkey@0"
I20260925 15:04:33.009733 26618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:33.024464 26618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:33.028105 26618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:33.028388 26618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:33.029196 26618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:33.030360 26618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:33.030426 26618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:33.030529 26618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:33.030582 26618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:33.058218 26618 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:32989
I20260925 15:04:33.058259 26745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:32989 every 8 connection(s)
I20260925 15:04:33.059403 26618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:04:33.067077 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 26618
I20260925 15:04:33.067253 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:04:33.068080 23472 external_mini_cluster.cc:1428] Setting key 2ede1cd7b1607423533be04d4bacc129
Sep 25 15:04:33 dist-test-slave-nsrc krb5kdc[26512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348672, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:33.078948 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:33.061786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:43709 (local address 127.22.236.62:36057)
0925 15:04:33.062201 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:33.062205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:33.063185 (+ 980us) server_negotiation.cc:407] Connection header received
0925 15:04:33.064978 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:33.064982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:33.065055 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:33.065140 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:33.066513 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:33.067228 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:33.069325 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:33.069875 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:33.069996 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:33.073243 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:33.073248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:33.073251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:33.073281 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:33.075867 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:33.076873 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:33.076877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:33.076879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:33.076995 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:33.077395 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:33.077398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:33.077400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:33.077642 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:33.077782 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:33.078529 (+ 747us) server_negotiation.cc:299] Negotiation successful
0925 15:04:33.078689 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":210,"threads_started":1}
I20260925 15:04:33.080293 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:33.062327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36057 (local address 127.22.236.1:43709)
0925 15:04:33.062981 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:33.063051 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:33.064683 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:33.065328 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:33.065336 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:33.065818 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:33.066381 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:33.066397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:33.067401 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:33.067406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:33.069202 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:33.069213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:33.070096 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:33.070102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:33.070321 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:33.070916 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:33.070939 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:33.073050 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:33.076018 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:33.076027 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:33.076068 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:33.076742 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:33.077085 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:33.077089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:33.077092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:33.077307 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:33.077743 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:33.077750 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:33.078147 (+ 397us) client_negotiation.cc:769] Sending connection context
0925 15:04:33.078427 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 15:04:33.078902 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":237,"threads_started":1}
I20260925 15:04:33.081835 26746 heartbeater.cc:347] Connected to a master server at 127.22.236.62:36057
I20260925 15:04:33.082105 26746 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:33.082772 26746 heartbeater.cc:511] Master 127.22.236.62:36057 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
I20260925 15:04:33.085043 26557 ts_manager.cc:194] Registered new tserver with Master: 3f92a15875a54a81b66e7e2f8e2182e2 (127.22.236.1:32989)
I20260925 15:04:33.086731 26557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:43709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:04:33.112779 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36057
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:35929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:33.413203 26752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:33.413488 26752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:33.413596 26752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:33.420641 26752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:33.420737 26752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:33.420850 26752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:04:33.430322 26752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:35929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:33.431488 26752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:33.432703 26752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:33.444394 26757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:33.444437 26758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:33.444481 26760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:33.445336 26752 server_base.cc:1034] running on GCE node
I20260925 15:04:33.445885 26752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:33.446734 26752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:33.447921 26752 hybrid_clock.cc:648] HybridClock initialized: now 1790348673447880 us; error 44 us; skew 500 ppm
I20260925 15:04:33.448231 26752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:33 dist-test-slave-nsrc krb5kdc[26512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348673, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:33.453935 26752 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:04:33.455231 26752 webserver.cc:460] Webserver started at http://127.22.236.2:39991/ using document root <none> and password file <none>
I20260925 15:04:33.455860 26752 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:33.455971 26752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:33.456275 26752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:33.458982 26752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "d7966efd18844806a7d08cdaf9320da1"
format_stamp: "Formatted at 2026-09-25 15:04:33 on dist-test-slave-nsrc"
server_key: "c1a240854c5b58ad8258a31f62c5fde2"
server_key_iv: "7e3815070d96c955cdb3d010c7e10489"
server_key_version: "encryptionkey@0"
I20260925 15:04:33.459707 26752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "d7966efd18844806a7d08cdaf9320da1"
format_stamp: "Formatted at 2026-09-25 15:04:33 on dist-test-slave-nsrc"
server_key: "c1a240854c5b58ad8258a31f62c5fde2"
server_key_iv: "7e3815070d96c955cdb3d010c7e10489"
server_key_version: "encryptionkey@0"
I20260925 15:04:33.464638 26752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:04:33.467757 26767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:33.469012 26752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:04:33.469142 26752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "d7966efd18844806a7d08cdaf9320da1"
format_stamp: "Formatted at 2026-09-25 15:04:33 on dist-test-slave-nsrc"
server_key: "c1a240854c5b58ad8258a31f62c5fde2"
server_key_iv: "7e3815070d96c955cdb3d010c7e10489"
server_key_version: "encryptionkey@0"
I20260925 15:04:33.469301 26752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:33.484822 26752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:33.489094 26752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:33.489382 26752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:33.490556 26752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:33.491839 26752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:33.491905 26752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:33.492022 26752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:33.492074 26752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:33.520078 26752 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:39311
I20260925 15:04:33.520117 26879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:39311 every 8 connection(s)
I20260925 15:04:33.521311 26752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:04:33.526906 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 26752
I20260925 15:04:33.527088 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:04:33.527633 23472 external_mini_cluster.cc:1428] Setting key eb886aaf66717287a872893548efd7c8
Sep 25 15:04:33 dist-test-slave-nsrc krb5kdc[26512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348673, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:33.540241 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:33.523550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:35747 (local address 127.22.236.62:36057)
0925 15:04:33.523714 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:33.523718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:33.524903 (+ 1185us) server_negotiation.cc:407] Connection header received
0925 15:04:33.526577 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:33.526582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:33.526644 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:33.526726 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:33.528584 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:33.529317 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:33.530782 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:33.531179 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:33.531285 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:33.534658 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:33.534664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:33.534669 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:33.534703 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:33.537521 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:33.538152 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:33.538157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:33.538160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:33.538300 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:33.538668 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:33.538673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:33.538676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:33.538978 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:33.539113 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:33.539705 (+ 592us) server_negotiation.cc:299] Negotiation successful
0925 15:04:33.539908 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:04:33.541105 26882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:33.524026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36057 (local address 127.22.236.2:35747)
0925 15:04:33.524760 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:33.524802 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:33.526307 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:33.526976 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:33.526988 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:33.527674 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:33.528423 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:33.528443 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:33.529465 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:33.529469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:33.530655 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:33.530666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:33.531384 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:33.531391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:33.531599 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:33.532237 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:33.532259 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:33.534469 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:33.537634 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:33.537641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:33.537653 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:33.538043 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:33.538388 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:33.538391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:33.538393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:33.538574 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:33.539076 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:33.539081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:33.539364 (+ 283us) client_negotiation.cc:769] Sending connection context
0925 15:04:33.539571 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 15:04:33.539952 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":321,"threads_started":1}
I20260925 15:04:33.542491 26880 heartbeater.cc:347] Connected to a master server at 127.22.236.62:36057
I20260925 15:04:33.542753 26880 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:33.543417 26880 heartbeater.cc:511] Master 127.22.236.62:36057 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
I20260925 15:04:33.544795 26557 ts_manager.cc:194] Registered new tserver with Master: d7966efd18844806a7d08cdaf9320da1 (127.22.236.2:39311)
I20260925 15:04:33.545600 26557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:35747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:04:33.565352 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36057
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:35929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:33.862206 26885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:33.862488 26885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:33.862622 26885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:33.869639 26885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:33.869748 26885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:33.869896 26885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:04:33.879269 26885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:35929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:33.880533 26885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:33.881790 26885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:33.893710 26890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:33.893923 26893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:33.893942 26891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:33.894505 26885 server_base.cc:1034] running on GCE node
I20260925 15:04:33.895027 26885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:33.895838 26885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:33.897037 26885 hybrid_clock.cc:648] HybridClock initialized: now 1790348673896990 us; error 36 us; skew 500 ppm
I20260925 15:04:33.897327 26885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:33 dist-test-slave-nsrc krb5kdc[26512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348673, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:33.900206 26885 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:04:33.901521 26885 webserver.cc:460] Webserver started at http://127.22.236.3:43507/ using document root <none> and password file <none>
I20260925 15:04:33.902125 26885 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:33.902218 26885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:33.902446 26885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:33.905159 26885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "97da4a3206624df1a5005975c9c4178d"
format_stamp: "Formatted at 2026-09-25 15:04:33 on dist-test-slave-nsrc"
server_key: "e75d3cd1dac52e552dc326f604bcfe5d"
server_key_iv: "906a69d50daf0e4ef54602e729786fe1"
server_key_version: "encryptionkey@0"
I20260925 15:04:33.905843 26885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "97da4a3206624df1a5005975c9c4178d"
format_stamp: "Formatted at 2026-09-25 15:04:33 on dist-test-slave-nsrc"
server_key: "e75d3cd1dac52e552dc326f604bcfe5d"
server_key_iv: "906a69d50daf0e4ef54602e729786fe1"
server_key_version: "encryptionkey@0"
I20260925 15:04:33.910566 26885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:04:33.913736 26900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:33.915022 26885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:04:33.915162 26885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "97da4a3206624df1a5005975c9c4178d"
format_stamp: "Formatted at 2026-09-25 15:04:33 on dist-test-slave-nsrc"
server_key: "e75d3cd1dac52e552dc326f604bcfe5d"
server_key_iv: "906a69d50daf0e4ef54602e729786fe1"
server_key_version: "encryptionkey@0"
I20260925 15:04:33.915320 26885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:33.927336 26885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:33.930771 26885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:33.931022 26885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:33.931834 26885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:33.933059 26885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:33.933132 26885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:33.933241 26885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:33.933295 26885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:33.962842 26885 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:37369
I20260925 15:04:33.962880 27012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:37369 every 8 connection(s)
I20260925 15:04:33.964072 26885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:04:33.968938 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 26885
I20260925 15:04:33.969094 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:04:33.969640 23472 external_mini_cluster.cc:1428] Setting key cd7716fbf0ef047f07e90cdc2e96d477
Sep 25 15:04:33 dist-test-slave-nsrc krb5kdc[26512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348673, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:33.982286 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:33.966435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:34937 (local address 127.22.236.62:36057)
0925 15:04:33.966591 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:33.966596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:33.967812 (+ 1216us) server_negotiation.cc:407] Connection header received
0925 15:04:33.970059 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:33.970063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:33.970134 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:33.970247 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:33.972057 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:33.972913 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:33.974384 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:33.974780 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:33.974876 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:33.977881 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:33.977887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:33.977890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:33.977918 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:33.980007 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:33.980632 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:33.980635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:33.980637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:33.980738 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:33.981063 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:33.981066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:33.981068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:33.981299 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:33.981397 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:33.981918 (+ 521us) server_negotiation.cc:299] Negotiation successful
0925 15:04:33.982060 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:04:33.983278 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:33.966934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36057 (local address 127.22.236.3:34937)
0925 15:04:33.967661 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:33.967706 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:33.969774 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:33.970463 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:33.970474 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:33.971145 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:33.971888 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:33.971924 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:33.973033 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:33.973037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:33.974274 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:33.974285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:33.974945 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:33.974949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:33.975122 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:33.975719 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:33.975740 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:33.977700 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:33.980119 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:33.980125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:33.980137 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:33.980542 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:33.980810 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:33.980813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:33.980815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:33.980984 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:33.981381 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:33.981385 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:33.981652 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 15:04:33.981847 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 15:04:33.982195 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":294,"threads_started":1}
I20260925 15:04:33.984655 27013 heartbeater.cc:347] Connected to a master server at 127.22.236.62:36057
I20260925 15:04:33.984953 27013 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:33.985576 27013 heartbeater.cc:511] Master 127.22.236.62:36057 requested a full tablet report, sending...
I20260925 15:04:33.986754 26556 ts_manager.cc:194] Registered new tserver with Master: 97da4a3206624df1a5005975c9c4178d (127.22.236.3:37369)
I20260925 15:04:33.987527 26556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:34937
I20260925 15:04:33.996603 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:04:34 dist-test-slave-nsrc krb5kdc[26512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348674, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 15:04:34 dist-test-slave-nsrc krb5kdc[26512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348674, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:34.038478 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:34.025586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.22.236.62:36057)
0925 15:04:34.025760 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:34.025767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:34.026130 (+ 363us) server_negotiation.cc:407] Connection header received
0925 15:04:34.026299 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:34.026303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:34.026369 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:34.026455 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:34.027776 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:34.028561 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:34.029739 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:34.030151 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:34.030263 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:34.033747 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:34.033753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:34.033756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:34.033787 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:34.035953 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:34.036540 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:34.036545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:34.036549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:34.036689 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:34.037069 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:34.037074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:34.037077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:34.037483 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:34.037692 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:34.037970 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 15:04:34.038166 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 15:04:34.038991 26556 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56094
I20260925 15:04:34.040663 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 26618
Sep 25 15:04:34 dist-test-slave-nsrc krb5kdc[26512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348674, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:04:34.061400 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:34.041582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.22.236.1:32989)
0925 15:04:34.048496 (+ 6914us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:34.048504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:34.048529 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:04:34.048581 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:34.048587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:34.048783 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:34.048939 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:34.050135 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:34.051077 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:34.052242 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:34.052744 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:34.052837 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:34.056217 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:34.056227 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:34.056241 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:34.056272 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:34.058776 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:34.059484 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:34.059490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:34.059493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:34.059598 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:34.059953 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:34.059956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:34.059958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:34.060322 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:34.060543 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:34.060821 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 15:04:34.061007 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6774,"thread_start_us":6676,"threads_started":1}
W20260925 15:04:34.062528 26741 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49440
W20260925 15:04:34.063588 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 26618: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:04:34.063825 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 26618
I20260925 15:04:34.075563 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 26752
Sep 25 15:04:34 dist-test-slave-nsrc krb5kdc[26512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348674, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:04:34.089805 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:34.076355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.22.236.2:39311)
0925 15:04:34.076854 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:34.076861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:34.076885 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:04:34.076958 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:34.076964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:34.077155 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:34.077284 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:34.078302 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:34.079255 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:34.080475 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:34.080998 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:34.081088 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:34.084742 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:34.084753 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:34.084767 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:34.084797 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:34.087215 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:34.087747 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:34.087753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:34.087755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:34.087860 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:34.088298 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:34.088303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:34.088305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:34.088648 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:34.088933 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:34.089249 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 15:04:34.089493 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":255,"threads_started":1}
W20260925 15:04:34.090958 26875 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53356
W20260925 15:04:34.091970 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 26752: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:04:34.092118 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 26752
I20260925 15:04:34.103421 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 26885
Sep 25 15:04:34 dist-test-slave-nsrc krb5kdc[26512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348674, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:04:34.117919 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:34.104309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.22.236.3:37369)
0925 15:04:34.104888 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:34.104894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:34.104921 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:04:34.104976 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:34.104982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:34.105177 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:34.105294 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:34.106473 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:34.107473 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:34.108674 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:34.109201 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:34.109291 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:34.113159 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:34.113170 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:34.113184 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:34.113217 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:34.115731 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:34.116300 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:34.116307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:34.116309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:34.116421 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:34.116682 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:34.116684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:34.116686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:34.116995 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:34.117210 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:34.117387 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 15:04:34.117566 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":277,"threads_started":1}
W20260925 15:04:34.119071 27008 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44020
W20260925 15:04:34.120165 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 26885: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:04:34.120316 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 26885
I20260925 15:04:34.131827 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 26525
I20260925 15:04:34.132889 26587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:34.231046 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 26525
2026-09-25T15:04:34Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3326 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:04:34 dist-test-slave-nsrc krb5kdc[27030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:04:34 dist-test-slave-nsrc krb5kdc[27030](info): set up 2 sockets
Sep 25 15:04:34 dist-test-slave-nsrc krb5kdc[27030](info): commencing operation
krb5kdc: starting...
W20260925 15:04:35.212318 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:04:35 dist-test-slave-nsrc krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:04:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:04:35.381860 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:33897
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:35.684653 27043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:35.684953 27043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:35.685077 27043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:35.692078 27043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:35.692209 27043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:35.692289 27043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:35.692368 27043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:35.692430 27043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:04:35.702052 27043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:33897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:33897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:35.703361 27043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:35.704619 27043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:35.715533 27048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:35.715579 27049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:35.715629 27051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:35.716325 27043 server_base.cc:1034] running on GCE node
I20260925 15:04:35.717103 27043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:35.718495 27043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:35.719707 27043 hybrid_clock.cc:648] HybridClock initialized: now 1790348675719657 us; error 34 us; skew 500 ppm
I20260925 15:04:35.720115 27043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:35 dist-test-slave-nsrc krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348675, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:35.723217 27043 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:04:35.724700 27043 webserver.cc:460] Webserver started at http://127.22.236.62:37093/ using document root <none> and password file <none>
I20260925 15:04:35.725360 27043 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:35.725471 27043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:35.725746 27043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:35.728709 27043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "dfb038ecefcd4eeb8bf439f23a06f10d"
format_stamp: "Formatted at 2026-09-25 15:04:35 on dist-test-slave-nsrc"
server_key: "c6f3f7eea3fb261063a1242ec0f450e9"
server_key_iv: "233c9c690df8192eb1fb8385266d20c9"
server_key_version: "encryptionkey@0"
I20260925 15:04:35.729432 27043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "dfb038ecefcd4eeb8bf439f23a06f10d"
format_stamp: "Formatted at 2026-09-25 15:04:35 on dist-test-slave-nsrc"
server_key: "c6f3f7eea3fb261063a1242ec0f450e9"
server_key_iv: "233c9c690df8192eb1fb8385266d20c9"
server_key_version: "encryptionkey@0"
I20260925 15:04:35.734194 27043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:04:35.737365 27058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:35.738551 27043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:35.738698 27043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "dfb038ecefcd4eeb8bf439f23a06f10d"
format_stamp: "Formatted at 2026-09-25 15:04:35 on dist-test-slave-nsrc"
server_key: "c6f3f7eea3fb261063a1242ec0f450e9"
server_key_iv: "233c9c690df8192eb1fb8385266d20c9"
server_key_version: "encryptionkey@0"
I20260925 15:04:35.738866 27043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:35.752321 27043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:35.756049 27043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:35.756351 27043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:35.772848 27043 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:33897
I20260925 15:04:35.772881 27109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:33897 every 8 connection(s)
I20260925 15:04:35.774123 27043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:35.777624 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 27043
I20260925 15:04:35.777738 27110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:35.777844 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:35.778597 23472 external_mini_cluster.cc:1428] Setting key ecd9ddc489d10c3a498b0e04eade7ac3
I20260925 15:04:35.790860 27110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d: Bootstrap starting.
Sep 25 15:04:35 dist-test-slave-nsrc krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:35.794380 27110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:35.795374 27110 log.cc:824] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:35.798457 27110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d: No bootstrap required, opened a new log
I20260925 15:04:35.801334 27113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:35.780573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.22.236.62:33897)
0925 15:04:35.781329 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:35.781339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:35.781379 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:04:35.783542 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:35.783566 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:35.783967 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:35.784314 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:35.785985 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:35.787212 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:35.788909 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:35.789544 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:35.789676 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:35.794049 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:35.794056 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:35.794070 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:35.794108 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:35.797336 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:35.798155 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:35.798162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:35.798168 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:35.798320 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:35.798706 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:35.798710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:35.798713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:35.799252 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:35.799481 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:35.799887 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 15:04:35.800284 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":374,"threads_started":1}
I20260925 15:04:35.806023 27110 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb038ecefcd4eeb8bf439f23a06f10d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33897 } }
I20260925 15:04:35.806344 27110 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:35.806444 27110 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb038ecefcd4eeb8bf439f23a06f10d, State: Initialized, Role: FOLLOWER
I20260925 15:04:35.806955 27110 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb038ecefcd4eeb8bf439f23a06f10d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33897 } }
I20260925 15:04:35.807147 27110 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:35.807258 27110 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:35.807411 27110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:35.809602 27110 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb038ecefcd4eeb8bf439f23a06f10d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33897 } }
I20260925 15:04:35.810045 27110 leader_election.cc:302] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfb038ecefcd4eeb8bf439f23a06f10d; no voters:
I20260925 15:04:35.810529 27110 leader_election.cc:288] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:35.810636 27115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:35.811209 27115 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [term 1 LEADER]: Becoming Leader. State: Replica: dfb038ecefcd4eeb8bf439f23a06f10d, State: Running, Role: LEADER
I20260925 15:04:35.811825 27115 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb038ecefcd4eeb8bf439f23a06f10d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33897 } }
I20260925 15:04:35.812565 27110 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:35.814615 27117 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfb038ecefcd4eeb8bf439f23a06f10d. Latest consensus state: current_term: 1 leader_uuid: "dfb038ecefcd4eeb8bf439f23a06f10d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb038ecefcd4eeb8bf439f23a06f10d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33897 } } }
I20260925 15:04:35.815054 27117 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:35.814941 27116 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfb038ecefcd4eeb8bf439f23a06f10d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb038ecefcd4eeb8bf439f23a06f10d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33897 } } }
I20260925 15:04:35.815303 27116 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:35.817204 27124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:35.822485 27124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:35.830673 27124 catalog_manager.cc:1347] Generated new cluster ID: d5319c52efea4832bf76bd0cf5ac92a3
I20260925 15:04:35.830804 27124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:35.839753 27124 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:35.841614 27124 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:35.849815 27124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfb038ecefcd4eeb8bf439f23a06f10d: Generated new TSK 0
I20260925 15:04:35.850852 27124 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:04:35.889142 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33897
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:36.195809 27136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:36.196080 27136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:36.196215 27136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:36.203341 27136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:36.203441 27136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:36.203557 27136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:04:36.212952 27136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:36.214186 27136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:36.215422 27136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:36.227660 27141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:36.227694 27144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:36.227723 27142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:36.228473 27136 server_base.cc:1034] running on GCE node
I20260925 15:04:36.229055 27136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:36.229997 27136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:36.231196 27136 hybrid_clock.cc:648] HybridClock initialized: now 1790348676231148 us; error 43 us; skew 500 ppm
I20260925 15:04:36.231479 27136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:36 dist-test-slave-nsrc krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348676, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:36.234529 27136 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:04:36.235853 27136 webserver.cc:460] Webserver started at http://127.22.236.1:36147/ using document root <none> and password file <none>
I20260925 15:04:36.236532 27136 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:36.236645 27136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:36.236937 27136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:36.239693 27136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "6aa5c9377a544056b1f27b0e8d1dd650"
format_stamp: "Formatted at 2026-09-25 15:04:36 on dist-test-slave-nsrc"
server_key: "56bc703e148b7187b65dc5092d7a77e3"
server_key_iv: "f1fe30b2b267369c4ea988bcd014fed6"
server_key_version: "encryptionkey@0"
I20260925 15:04:36.240514 27136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "6aa5c9377a544056b1f27b0e8d1dd650"
format_stamp: "Formatted at 2026-09-25 15:04:36 on dist-test-slave-nsrc"
server_key: "56bc703e148b7187b65dc5092d7a77e3"
server_key_iv: "f1fe30b2b267369c4ea988bcd014fed6"
server_key_version: "encryptionkey@0"
I20260925 15:04:36.245492 27136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 15:04:36.248725 27151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:36.249990 27136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:04:36.250130 27136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "6aa5c9377a544056b1f27b0e8d1dd650"
format_stamp: "Formatted at 2026-09-25 15:04:36 on dist-test-slave-nsrc"
server_key: "56bc703e148b7187b65dc5092d7a77e3"
server_key_iv: "f1fe30b2b267369c4ea988bcd014fed6"
server_key_version: "encryptionkey@0"
I20260925 15:04:36.250303 27136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:36.273461 27136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:36.277668 27136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:36.277949 27136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:36.278913 27136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:36.280232 27136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:36.280308 27136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:36.280424 27136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:36.280476 27136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:36.310053 27136 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:46455
I20260925 15:04:36.310123 27263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:46455 every 8 connection(s)
I20260925 15:04:36.311264 27136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:04:36.315127 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 27136
I20260925 15:04:36.315346 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:04:36.316121 23472 external_mini_cluster.cc:1428] Setting key 7c965a143ea15bad9c77ef2307505dc9
Sep 25 15:04:36 dist-test-slave-nsrc krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348676, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:36.332654 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:36.313874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:52347 (local address 127.22.236.62:33897)
0925 15:04:36.314265 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:36.314271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:36.316205 (+ 1934us) server_negotiation.cc:407] Connection header received
0925 15:04:36.317728 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:36.317733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:36.317800 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:36.317911 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:36.319782 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:36.320536 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:36.322578 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:36.322962 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:36.323050 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:36.326501 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:36.326508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:36.326512 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:36.326549 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:36.329546 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:36.330327 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:36.330333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:36.330337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:36.330477 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:36.330896 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:36.330901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:36.330904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:36.331238 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:36.331404 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:36.332040 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 15:04:36.332274 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":176,"threads_started":1}
I20260925 15:04:36.333508 27267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:36.314386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33897 (local address 127.22.236.1:52347)
0925 15:04:36.315231 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:36.315327 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:36.317485 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:36.318132 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:36.318143 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:36.318849 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:36.319608 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:36.319632 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:36.320689 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:36.320693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:36.322442 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:36.322456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:36.323163 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:36.323169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:36.323439 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:36.324041 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:36.324064 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:36.326296 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:36.329676 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:36.329683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:36.329696 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:36.330215 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:36.330584 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:36.330587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:36.330589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:36.330793 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:36.331354 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:36.331359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:36.331666 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 15:04:36.331932 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 15:04:36.332355 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":335,"threads_started":1}
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
I20260925 15:04:36.334980 27264 heartbeater.cc:347] Connected to a master server at 127.22.236.62:33897
I20260925 15:04:36.335247 27264 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
I20260925 15:04:36.335943 27264 heartbeater.cc:511] Master 127.22.236.62:33897 requested a full tablet report, sending...
I20260925 15:04:36.338338 27075 ts_manager.cc:194] Registered new tserver with Master: 6aa5c9377a544056b1f27b0e8d1dd650 (127.22.236.1:46455)
I20260925 15:04:36.340271 27075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:52347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:04:36.357420 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33897
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:36.658196 27270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:36.658473 27270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:36.658564 27270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:36.665619 27270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:36.665719 27270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:36.665884 27270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:04:36.675380 27270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:36.676594 27270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:36.677799 27270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:36.689831 27276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:36.690023 27275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:36.689935 27278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:36.690932 27270 server_base.cc:1034] running on GCE node
I20260925 15:04:36.691483 27270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:36.692397 27270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:36.693585 27270 hybrid_clock.cc:648] HybridClock initialized: now 1790348676693547 us; error 34 us; skew 500 ppm
I20260925 15:04:36.693876 27270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:36 dist-test-slave-nsrc krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348676, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:36.696785 27270 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:04:36.698138 27270 webserver.cc:460] Webserver started at http://127.22.236.2:37879/ using document root <none> and password file <none>
I20260925 15:04:36.698742 27270 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:36.698834 27270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:36.699102 27270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:36.701824 27270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "bf58c6bbaaff4276bc7d09547b67a784"
format_stamp: "Formatted at 2026-09-25 15:04:36 on dist-test-slave-nsrc"
server_key: "8bbc173521b76ee1f5f9fc66d4805372"
server_key_iv: "e70aa7d3e0dde8845463a06c2ab00041"
server_key_version: "encryptionkey@0"
I20260925 15:04:36.702504 27270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "bf58c6bbaaff4276bc7d09547b67a784"
format_stamp: "Formatted at 2026-09-25 15:04:36 on dist-test-slave-nsrc"
server_key: "8bbc173521b76ee1f5f9fc66d4805372"
server_key_iv: "e70aa7d3e0dde8845463a06c2ab00041"
server_key_version: "encryptionkey@0"
I20260925 15:04:36.707233 27270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:04:36.710439 27285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:36.711663 27270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:04:36.711805 27270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "bf58c6bbaaff4276bc7d09547b67a784"
format_stamp: "Formatted at 2026-09-25 15:04:36 on dist-test-slave-nsrc"
server_key: "8bbc173521b76ee1f5f9fc66d4805372"
server_key_iv: "e70aa7d3e0dde8845463a06c2ab00041"
server_key_version: "encryptionkey@0"
I20260925 15:04:36.711995 27270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:36.734441 27270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:36.738274 27270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:36.738533 27270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:36.739444 27270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:36.740705 27270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:36.740782 27270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:36.740901 27270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:36.740952 27270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:36.769939 27270 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:41949
I20260925 15:04:36.770002 27397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:41949 every 8 connection(s)
I20260925 15:04:36.771152 27270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:04:36.773140 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 27270
I20260925 15:04:36.773320 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:04:36.773860 23472 external_mini_cluster.cc:1428] Setting key a1963d1f0b9d44cbdfd3d64cfeaa7958
Sep 25 15:04:36 dist-test-slave-nsrc krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348676, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
I20260925 15:04:36.793242 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:36.773632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:60285 (local address 127.22.236.62:33897)
0925 15:04:36.773826 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:36.773831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:36.775180 (+ 1349us) server_negotiation.cc:407] Connection header received
0925 15:04:36.777627 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:36.777633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:36.777726 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:36.777825 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:36.779661 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:36.780796 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:36.782457 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:36.782948 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:36.783061 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:36.787110 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:36.787117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:36.787123 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:36.787160 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:36.790338 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:36.791052 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:36.791058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:36.791061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:36.791219 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:36.791592 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:36.791596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:36.791600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:36.791921 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:36.792066 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:36.792715 (+ 649us) server_negotiation.cc:299] Negotiation successful
0925 15:04:36.792933 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:04:36.794068 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:36.774209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33897 (local address 127.22.236.2:60285)
0925 15:04:36.775035 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:36.775082 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:36.777331 (+ 2249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:36.778070 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:36.778082 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:36.778742 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:36.779478 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:36.779506 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:36.780974 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:36.780979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:36.782316 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:36.782328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:36.783159 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:36.783165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:36.783408 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:36.784074 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:36.784102 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:36.786884 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:36.790500 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:36.790507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:36.790520 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:36.790938 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:36.791321 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:36.791324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:36.791326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:36.791495 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:36.792025 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:36.792030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:36.792366 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 15:04:36.792585 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 15:04:36.792990 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":309,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
I20260925 15:04:36.795447 27398 heartbeater.cc:347] Connected to a master server at 127.22.236.62:33897
I20260925 15:04:36.795743 27398 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:36.796429 27398 heartbeater.cc:511] Master 127.22.236.62:33897 requested a full tablet report, sending...
I20260925 15:04:36.797712 27075 ts_manager.cc:194] Registered new tserver with Master: bf58c6bbaaff4276bc7d09547b67a784 (127.22.236.2:41949)
I20260925 15:04:36.798476 27075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:60285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:04:36.815238 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33897
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:37.126178 27403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:37.126451 27403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:37.126572 27403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:37.133471 27403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:37.133580 27403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:37.133728 27403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:04:37.143002 27403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:37.144168 27403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:37.145413 27403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:37.157346 27408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:37.157577 27409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:37.157994 27411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:37.158432 27403 server_base.cc:1034] running on GCE node
I20260925 15:04:37.158948 27403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:37.159793 27403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:37.160975 27403 hybrid_clock.cc:648] HybridClock initialized: now 1790348677160931 us; error 37 us; skew 500 ppm
I20260925 15:04:37.161254 27403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:37 dist-test-slave-nsrc krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348677, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:37.164047 27403 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:04:37.165413 27403 webserver.cc:460] Webserver started at http://127.22.236.3:38035/ using document root <none> and password file <none>
I20260925 15:04:37.166061 27403 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:37.166170 27403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:37.166441 27403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:37.169251 27403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "138dadce50514310bca86cceb4acbf3a"
format_stamp: "Formatted at 2026-09-25 15:04:37 on dist-test-slave-nsrc"
server_key: "135cdecbae7fec3c0f9b39eb88c372d9"
server_key_iv: "1b2d1bc861a1c7f0d82064ab678b7478"
server_key_version: "encryptionkey@0"
I20260925 15:04:37.169981 27403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "138dadce50514310bca86cceb4acbf3a"
format_stamp: "Formatted at 2026-09-25 15:04:37 on dist-test-slave-nsrc"
server_key: "135cdecbae7fec3c0f9b39eb88c372d9"
server_key_iv: "1b2d1bc861a1c7f0d82064ab678b7478"
server_key_version: "encryptionkey@0"
I20260925 15:04:37.174759 27403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:04:37.177987 27418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:37.179180 27403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:04:37.179320 27403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "138dadce50514310bca86cceb4acbf3a"
format_stamp: "Formatted at 2026-09-25 15:04:37 on dist-test-slave-nsrc"
server_key: "135cdecbae7fec3c0f9b39eb88c372d9"
server_key_iv: "1b2d1bc861a1c7f0d82064ab678b7478"
server_key_version: "encryptionkey@0"
I20260925 15:04:37.179489 27403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:37.199659 27403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:37.203312 27403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:37.203557 27403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:37.204444 27403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:37.205677 27403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:37.205749 27403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:37.205874 27403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:37.205927 27403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:37.234669 27403 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:39043
I20260925 15:04:37.234714 27530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:39043 every 8 connection(s)
I20260925 15:04:37.235812 27403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:04:37.240988 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 27403
I20260925 15:04:37.241145 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:04:37.241533 23472 external_mini_cluster.cc:1428] Setting key 3976f4e18455c61625b113c1a2e958f3
Sep 25 15:04:37 dist-test-slave-nsrc krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348677, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:37.254446 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:37.238132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:57283 (local address 127.22.236.62:33897)
0925 15:04:37.238299 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:37.238303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:37.239402 (+ 1099us) server_negotiation.cc:407] Connection header received
0925 15:04:37.241189 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:37.241194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:37.241278 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:37.241361 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:37.242994 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:37.243794 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:37.245371 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:37.245786 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:37.245892 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:37.249163 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:37.249168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:37.249171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:37.249207 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:37.251621 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:37.252300 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:37.252304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:37.252306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:37.252415 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:37.252805 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:37.252809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:37.252811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:37.253062 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:37.253183 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:37.253878 (+ 695us) server_negotiation.cc:299] Negotiation successful
0925 15:04:37.254090 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:04:37.255342 27533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:37.238609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33897 (local address 127.22.236.3:57283)
0925 15:04:37.239247 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:37.239288 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:37.240933 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:37.241598 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:37.241607 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:37.242195 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:37.242833 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:37.242852 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:37.243930 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:37.243934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:37.245241 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:37.245253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:37.245990 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:37.245997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:37.246213 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:37.246798 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:37.246823 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:37.248966 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:37.251749 (+ 2783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:37.251759 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:37.251775 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:37.252189 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:37.252521 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:37.252525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:37.252529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:37.252697 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:37.253175 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:37.253182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:37.253532 (+ 350us) client_negotiation.cc:769] Sending connection context
0925 15:04:37.253773 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 15:04:37.254243 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":239,"threads_started":1}
I20260925 15:04:37.256641 27531 heartbeater.cc:347] Connected to a master server at 127.22.236.62:33897
I20260925 15:04:37.256903 27531 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:37.257541 27531 heartbeater.cc:511] Master 127.22.236.62:33897 requested a full tablet report, sending...
I20260925 15:04:37.258728 27075 ts_manager.cc:194] Registered new tserver with Master: 138dadce50514310bca86cceb4acbf3a (127.22.236.3:39043)
I20260925 15:04:37.259477 27075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:57283
I20260925 15:04:37.268181 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:04:37 dist-test-slave-nsrc krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:04:37 dist-test-slave-nsrc krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:04:37.310716 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:37.296429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.22.236.1:46455)
0925 15:04:37.297224 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:37.297233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:37.297280 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:04:37.297350 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:37.297359 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:37.297566 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:37.297705 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:37.299118 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:37.300072 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:37.301276 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:37.301797 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:37.301907 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:37.305468 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:37.305478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:37.305491 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:37.305531 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:37.308107 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:37.308725 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:37.308732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:37.308734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:37.308857 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:37.309205 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:37.309209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:37.309211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:37.309559 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:37.309863 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:37.310127 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 15:04:37.310376 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":489,"threads_started":1}
Sep 25 15:04:37 dist-test-slave-nsrc krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:37.327482 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:37.315190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.22.236.62:33897)
0925 15:04:37.315379 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:37.315384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:37.315613 (+ 229us) server_negotiation.cc:407] Connection header received
0925 15:04:37.315765 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:37.315769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:37.315828 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:37.315902 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:37.317104 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:37.317821 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:37.318940 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:37.319368 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:37.319457 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:37.322846 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:37.322851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:37.322854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:37.322885 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:37.325168 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:37.325733 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:37.325737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:37.325746 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:37.325874 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:37.326187 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:37.326192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:37.326194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:37.326472 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:37.326686 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:37.326993 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 15:04:37.327187 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 15:04:37.328097 27075 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42130
I20260925 15:04:37.328962 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 27136
I20260925 15:04:37.340404 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:37.329865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.22.236.1:46455)
0925 15:04:37.330045 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:37.330050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:37.330235 (+ 185us) server_negotiation.cc:407] Connection header received
0925 15:04:37.330378 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:37.330382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:37.330465 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:37.330542 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:37.331622 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:37.332318 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:37.333467 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:37.333864 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:37.333971 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:37.335919 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:37.335923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:37.335927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:37.335961 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:37.338139 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:37.338720 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:37.338724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:37.338726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:37.338833 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:37.339145 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:37.339150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:37.339152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:37.339391 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:37.339590 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:37.339877 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 15:04:37.340075 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:04:37.340950 27259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:37.420436 27264 heartbeater.cc:503] Master 127.22.236.62:33897 was elected leader, sending a full tablet report...
I20260925 15:04:37.421020 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 27136
W20260925 15:04:37.433001 27061 connection.cc:535] server connection from 127.22.236.1:52347 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:04:37.433149 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 27270
Sep 25 15:04:37 dist-test-slave-nsrc krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:04:37.447484 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:37.433968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.22.236.2:41949)
0925 15:04:37.434527 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:37.434534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:37.434559 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:04:37.434616 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:37.434622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:37.434841 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:37.434966 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:37.436020 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:37.437072 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:37.438193 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:37.438714 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:37.438810 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:37.442662 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:37.442672 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:37.442686 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:37.442722 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:37.445313 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:37.445850 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:37.445857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:37.445859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:37.445972 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:37.446231 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:37.446234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:37.446236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:37.446534 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:37.446794 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:37.446954 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:04:37.447149 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":292,"threads_started":1}
I20260925 15:04:37.448702 27393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:37.529330 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 27270
I20260925 15:04:37.541293 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 27403
Sep 25 15:04:37 dist-test-slave-nsrc krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:04:37.556389 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:37.542195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.22.236.3:39043)
0925 15:04:37.542823 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:37.542830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:37.542858 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:04:37.542916 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:37.542922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:37.543121 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:37.543292 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:37.544564 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:37.545578 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:37.546849 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:37.547389 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:37.547499 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:37.551525 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:37.551535 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:37.551550 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:37.551585 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:37.554203 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:37.554759 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:37.554765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:37.554767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:37.554888 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:37.555156 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:37.555159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:37.555161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:37.555464 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:37.555670 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:37.555888 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 15:04:37.556069 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":321,"threads_started":1}
I20260925 15:04:37.557533 27526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:37.637343 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 27403
I20260925 15:04:37.649143 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 27043
I20260925 15:04:37.650261 27105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:37.748907 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 27043
2026-09-25T15:04:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3518 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:04:37 dist-test-slave-nsrc krb5kdc[27550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:04:37 dist-test-slave-nsrc krb5kdc[27550](info): set up 2 sockets
Sep 25 15:04:37 dist-test-slave-nsrc krb5kdc[27550](info): commencing operation
krb5kdc: starting...
W20260925 15:04:38.732389 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:04:38 dist-test-slave-nsrc krb5kdc[27550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:04:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:04:38.901593 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:46507
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:35811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:46507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:39.199059 27563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:39.199344 27563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:39.199484 27563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:39.206568 27563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:39.206677 27563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:39.206753 27563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:39.206828 27563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:39.206900 27563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:04:39.216387 27563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:35811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:46507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:46507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:39.217581 27563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:39.218786 27563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:39.229228 27568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:39.229276 27569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:39.229295 27571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:39.230224 27563 server_base.cc:1034] running on GCE node
I20260925 15:04:39.230949 27563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:39.232273 27563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:39.233461 27563 hybrid_clock.cc:648] HybridClock initialized: now 1790348679233422 us; error 36 us; skew 500 ppm
I20260925 15:04:39.233760 27563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:39 dist-test-slave-nsrc krb5kdc[27550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348679, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:39.242650 27563 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:04:39.243716 27563 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:39.243799 27563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:39.244059 27563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:39.246802 27563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "135ed50a2e6748019485827df97e5598"
format_stamp: "Formatted at 2026-09-25 15:04:39 on dist-test-slave-nsrc"
server_key: "79f921ba9bc3feb893e2817a920b95a1"
server_key_iv: "21350c018f86629a0720c1af985f6982"
server_key_version: "encryptionkey@0"
I20260925 15:04:39.247495 27563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "135ed50a2e6748019485827df97e5598"
format_stamp: "Formatted at 2026-09-25 15:04:39 on dist-test-slave-nsrc"
server_key: "79f921ba9bc3feb893e2817a920b95a1"
server_key_iv: "21350c018f86629a0720c1af985f6982"
server_key_version: "encryptionkey@0"
I20260925 15:04:39.251971 27563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 15:04:39.255044 27577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:39.256268 27563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:04:39.256407 27563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "135ed50a2e6748019485827df97e5598"
format_stamp: "Formatted at 2026-09-25 15:04:39 on dist-test-slave-nsrc"
server_key: "79f921ba9bc3feb893e2817a920b95a1"
server_key_iv: "21350c018f86629a0720c1af985f6982"
server_key_version: "encryptionkey@0"
I20260925 15:04:39.256584 27563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:39.291406 27563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:39.295352 27563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:39.295593 27563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:39.311681 27563 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:46507
I20260925 15:04:39.311658 27628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:46507 every 8 connection(s)
I20260925 15:04:39.312537 27563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:39.315948 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:39.316427 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 27563
I20260925 15:04:39.316638 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:39.317389 23472 external_mini_cluster.cc:1428] Setting key 53d30b90b1e9d492b9c8ab50b821bf8b
I20260925 15:04:39.329025 27629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598: Bootstrap starting.
Sep 25 15:04:39 dist-test-slave-nsrc krb5kdc[27550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:39.332356 27629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:39.333410 27629 log.cc:824] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:39.335856 27629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598: No bootstrap required, opened a new log
I20260925 15:04:39.339049 27632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:39.319376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.22.236.62:46507)
0925 15:04:39.320096 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:39.320110 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:39.320172 (+ 62us) server_negotiation.cc:407] Connection header received
0925 15:04:39.321909 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:39.321929 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:39.322293 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:39.322614 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:39.324267 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:39.325515 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:39.327222 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:39.327830 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:39.327945 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:39.332123 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:39.332131 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:39.332170 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:39.332214 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:39.334965 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:39.335552 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:39.335558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:39.335564 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:39.335712 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:39.336060 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:39.336065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:39.336069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:39.336761 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:39.337032 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:39.337324 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:04:39.337727 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":334,"threads_started":1}
I20260925 15:04:39.343839 27629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ed50a2e6748019485827df97e5598" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 46507 } }
I20260925 15:04:39.344132 27629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:39.344233 27629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135ed50a2e6748019485827df97e5598, State: Initialized, Role: FOLLOWER
I20260925 15:04:39.344722 27629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ed50a2e6748019485827df97e5598" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 46507 } }
I20260925 15:04:39.344933 27629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:39.345021 27629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:39.345134 27629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:39.347055 27629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ed50a2e6748019485827df97e5598" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 46507 } }
I20260925 15:04:39.347484 27629 leader_election.cc:302] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 135ed50a2e6748019485827df97e5598; no voters:
I20260925 15:04:39.347978 27629 leader_election.cc:288] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:39.348078 27634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:39.348651 27634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [term 1 LEADER]: Becoming Leader. State: Replica: 135ed50a2e6748019485827df97e5598, State: Running, Role: LEADER
I20260925 15:04:39.349149 27634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ed50a2e6748019485827df97e5598" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 46507 } }
I20260925 15:04:39.349800 27629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:39.351953 27635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "135ed50a2e6748019485827df97e5598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ed50a2e6748019485827df97e5598" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 46507 } } }
I20260925 15:04:39.351946 27636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135ed50a2e6748019485827df97e5598. Latest consensus state: current_term: 1 leader_uuid: "135ed50a2e6748019485827df97e5598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ed50a2e6748019485827df97e5598" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 46507 } } }
I20260925 15:04:39.352371 27635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:39.352368 27636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:39.353191 27639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:39.359019 27639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:39.368378 27639 catalog_manager.cc:1347] Generated new cluster ID: d0a4d585252647fb8dcad1e585cf6f0e
I20260925 15:04:39.368496 27639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:39.387037 27639 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:39.388984 27639 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:39.400795 27639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598: Generated new TSK 0
I20260925 15:04:39.401834 27639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:04:39.447135 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:46507
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:35811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:39.754963 27655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:39.755255 27655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:39.755393 27655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:39.762600 27655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:39.762701 27655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:39.762846 27655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:04:39.772243 27655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:35811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:46507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:39.773496 27655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:39.774760 27655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:39.786826 27660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:39.786860 27661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:39.786788 27663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:39.787551 27655 server_base.cc:1034] running on GCE node
I20260925 15:04:39.788134 27655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:39.789019 27655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:39.790218 27655 hybrid_clock.cc:648] HybridClock initialized: now 1790348679790166 us; error 41 us; skew 500 ppm
I20260925 15:04:39.790531 27655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:39 dist-test-slave-nsrc krb5kdc[27550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348679, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:39.793397 27655 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:04:39.794440 27655 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:39.794534 27655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:39.794821 27655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:39.797549 27655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "e04c180a83bd449dae45fdea321b891d"
format_stamp: "Formatted at 2026-09-25 15:04:39 on dist-test-slave-nsrc"
server_key: "ecec1b6e3ed7f66aa123feca7ca4b2b8"
server_key_iv: "7c48d819cd20ccc598defc58c99e8594"
server_key_version: "encryptionkey@0"
I20260925 15:04:39.798275 27655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "e04c180a83bd449dae45fdea321b891d"
format_stamp: "Formatted at 2026-09-25 15:04:39 on dist-test-slave-nsrc"
server_key: "ecec1b6e3ed7f66aa123feca7ca4b2b8"
server_key_iv: "7c48d819cd20ccc598defc58c99e8594"
server_key_version: "encryptionkey@0"
I20260925 15:04:39.802904 27655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:04:39.806080 27669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:39.807317 27655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:04:39.807459 27655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "e04c180a83bd449dae45fdea321b891d"
format_stamp: "Formatted at 2026-09-25 15:04:39 on dist-test-slave-nsrc"
server_key: "ecec1b6e3ed7f66aa123feca7ca4b2b8"
server_key_iv: "7c48d819cd20ccc598defc58c99e8594"
server_key_version: "encryptionkey@0"
I20260925 15:04:39.807644 27655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:39.824301 27655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:39.828228 27655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:39.828474 27655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:39.829408 27655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:39.830637 27655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:39.830725 27655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:39.830856 27655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:39.830926 27655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:39.859993 27655 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:34793
I20260925 15:04:39.860054 27781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:34793 every 8 connection(s)
I20260925 15:04:39.860845 27655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:04:39.864065 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 27655
I20260925 15:04:39.864298 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:04:39.865018 23472 external_mini_cluster.cc:1428] Setting key c6c6314414fddc408b09d4e0568e9892
Sep 25 15:04:39 dist-test-slave-nsrc krb5kdc[27550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348679, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:39.880628 27784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:39.863061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:38091 (local address 127.22.236.62:46507)
0925 15:04:39.863467 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:39.863471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:39.864499 (+ 1028us) server_negotiation.cc:407] Connection header received
0925 15:04:39.866379 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:39.866385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:39.866463 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:39.866564 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:39.868445 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:39.869139 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:39.871123 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:39.871506 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:39.871593 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:39.874875 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:39.874881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:39.874886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:39.874921 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:39.877780 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:39.878400 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:39.878406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:39.878409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:39.878540 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:39.878934 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:39.878939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:39.878942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:39.879268 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:39.879412 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:39.880007 (+ 595us) server_negotiation.cc:299] Negotiation successful
0925 15:04:39.880237 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":215,"threads_started":1}
I20260925 15:04:39.881287 27785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:39.863590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:46507 (local address 127.22.236.1:38091)
0925 15:04:39.864336 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:39.864395 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:39.866132 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:39.866786 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:39.866797 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:39.867502 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:39.868276 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:39.868307 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:39.869275 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:39.869280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:39.870991 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:39.871005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:39.871699 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:39.871705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:39.871952 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:39.872592 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:39.872613 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:39.874674 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:39.877893 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:39.877902 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:39.877914 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:39.878290 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:39.878647 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:39.878651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:39.878654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:39.878833 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:39.879368 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:39.879373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:39.879667 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 15:04:39.879873 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 15:04:39.880267 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":288,"threads_started":1}
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
I20260925 15:04:39.882663 27782 heartbeater.cc:347] Connected to a master server at 127.22.236.62:46507
I20260925 15:04:39.882920 27782 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:39.883550 27782 heartbeater.cc:511] Master 127.22.236.62:46507 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
I20260925 15:04:39.885605 27594 ts_manager.cc:194] Registered new tserver with Master: e04c180a83bd449dae45fdea321b891d (127.22.236.1:34793)
I20260925 15:04:39.887486 27594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:38091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:04:39.906747 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:46507
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:35811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:40.205278 27788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:40.205549 27788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:40.205677 27788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:40.213215 27788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:40.213328 27788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:40.213485 27788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:04:40.222949 27788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:35811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:46507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:40.224179 27788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:40.225433 27788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:40.237237 27793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:40.237263 27794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:40.237305 27796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:40.237970 27788 server_base.cc:1034] running on GCE node
I20260925 15:04:40.238559 27788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:40.239436 27788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:40.240629 27788 hybrid_clock.cc:648] HybridClock initialized: now 1790348680240566 us; error 51 us; skew 500 ppm
I20260925 15:04:40.240932 27788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:40 dist-test-slave-nsrc krb5kdc[27550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348680, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:40.243670 27788 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:04:40.244712 27788 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:40.244822 27788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:40.245101 27788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:40.247835 27788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "f2326db6f66a4c0181140f3a9d68db81"
format_stamp: "Formatted at 2026-09-25 15:04:40 on dist-test-slave-nsrc"
server_key: "b9f2b6de35c247315b4e2bfc6ca6824c"
server_key_iv: "81b3985ef676dafeb469fee1cf336a18"
server_key_version: "encryptionkey@0"
I20260925 15:04:40.248589 27788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "f2326db6f66a4c0181140f3a9d68db81"
format_stamp: "Formatted at 2026-09-25 15:04:40 on dist-test-slave-nsrc"
server_key: "b9f2b6de35c247315b4e2bfc6ca6824c"
server_key_iv: "81b3985ef676dafeb469fee1cf336a18"
server_key_version: "encryptionkey@0"
I20260925 15:04:40.253270 27788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:04:40.256472 27802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:40.257791 27788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:04:40.257941 27788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "f2326db6f66a4c0181140f3a9d68db81"
format_stamp: "Formatted at 2026-09-25 15:04:40 on dist-test-slave-nsrc"
server_key: "b9f2b6de35c247315b4e2bfc6ca6824c"
server_key_iv: "81b3985ef676dafeb469fee1cf336a18"
server_key_version: "encryptionkey@0"
I20260925 15:04:40.258136 27788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:40.281947 27788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:40.285902 27788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:40.286131 27788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:40.287041 27788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:40.288338 27788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:40.288425 27788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:40.288525 27788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:40.288599 27788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:40.318531 27788 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:33611
I20260925 15:04:40.318571 27914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:33611 every 8 connection(s)
I20260925 15:04:40.319311 27788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:04:40.321130 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 27788
I20260925 15:04:40.321300 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:04:40.321805 23472 external_mini_cluster.cc:1428] Setting key 93d89cf41fe86d1b716401d6468ca866
Sep 25 15:04:40 dist-test-slave-nsrc krb5kdc[27550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348680, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
I20260925 15:04:40.344252 27784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.321689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:40171 (local address 127.22.236.62:46507)
0925 15:04:40.322664 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:40.322668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:40.328720 (+ 6052us) server_negotiation.cc:407] Connection header received
0925 15:04:40.330795 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:40.330801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:40.330887 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:40.331003 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:40.332446 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.333453 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.334895 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.335504 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.335614 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.338726 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:40.338731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:40.338735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:40.338766 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:40.341398 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:40.342156 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:40.342162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:40.342166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:40.342295 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:40.342681 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:40.342685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:40.342687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:40.342931 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:40.343054 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:40.343704 (+ 650us) server_negotiation.cc:299] Negotiation successful
0925 15:04:40.343912 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856}
I20260925 15:04:40.345242 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.322213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:46507 (local address 127.22.236.2:40171)
0925 15:04:40.328540 (+ 6327us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:40.328597 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:40.330524 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:40.331184 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:40.331193 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:40.331690 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:40.332292 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:40.332311 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.333589 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.333593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:40.334771 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:40.334782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.335693 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.335697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:40.335873 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.336497 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:40.336518 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:40.338514 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:40.341524 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:40.341533 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:40.341550 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:40.342041 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:40.342393 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:40.342398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:40.342402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:40.342575 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:40.343052 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:40.343060 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:40.343378 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 15:04:40.343597 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 15:04:40.344111 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6036,"thread_start_us":5841,"threads_started":1}
I20260925 15:04:40.346650 27915 heartbeater.cc:347] Connected to a master server at 127.22.236.62:46507
I20260925 15:04:40.346935 27915 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:40.347544 27915 heartbeater.cc:511] Master 127.22.236.62:46507 requested a full tablet report, sending...
I20260925 15:04:40.348831 27594 ts_manager.cc:194] Registered new tserver with Master: f2326db6f66a4c0181140f3a9d68db81 (127.22.236.2:33611)
I20260925 15:04:40.349583 27594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:40171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:04:40.361493 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:46507
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:35811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:40.661525 27920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:40.661791 27920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:40.661911 27920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:40.668962 27920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:40.669059 27920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:40.669198 27920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:04:40.678668 27920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:35811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:46507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:40.679919 27920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:40.681185 27920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:40.693164 27925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:40.693121 27928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:40.693209 27926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:40.694108 27920 server_base.cc:1034] running on GCE node
I20260925 15:04:40.694712 27920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:40.695600 27920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:40.696794 27920 hybrid_clock.cc:648] HybridClock initialized: now 1790348680696725 us; error 63 us; skew 500 ppm
I20260925 15:04:40.697083 27920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:40 dist-test-slave-nsrc krb5kdc[27550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348680, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:40.699928 27920 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:04:40.701027 27920 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:40.701124 27920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:40.701404 27920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:40.704134 27920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "f46d54b1aeeb4817ad111bac4f054771"
format_stamp: "Formatted at 2026-09-25 15:04:40 on dist-test-slave-nsrc"
server_key: "7bdf8a49f9dd30b666c654705d765a4c"
server_key_iv: "a63c1b775140de2a91ec0e48507c9962"
server_key_version: "encryptionkey@0"
I20260925 15:04:40.704932 27920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "f46d54b1aeeb4817ad111bac4f054771"
format_stamp: "Formatted at 2026-09-25 15:04:40 on dist-test-slave-nsrc"
server_key: "7bdf8a49f9dd30b666c654705d765a4c"
server_key_iv: "a63c1b775140de2a91ec0e48507c9962"
server_key_version: "encryptionkey@0"
I20260925 15:04:40.709600 27920 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:04:40.712813 27934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:40.714012 27920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:04:40.714159 27920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "f46d54b1aeeb4817ad111bac4f054771"
format_stamp: "Formatted at 2026-09-25 15:04:40 on dist-test-slave-nsrc"
server_key: "7bdf8a49f9dd30b666c654705d765a4c"
server_key_iv: "a63c1b775140de2a91ec0e48507c9962"
server_key_version: "encryptionkey@0"
I20260925 15:04:40.714339 27920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:40.732900 27920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:40.736892 27920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:40.737130 27920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:40.738014 27920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:40.739240 27920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:40.739328 27920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:40.739432 27920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:40.739507 27920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:40.769065 27920 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:43825
I20260925 15:04:40.769115 28046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:43825 every 8 connection(s)
I20260925 15:04:40.769840 27920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:04:40.777026 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 27920
I20260925 15:04:40.777226 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:04:40.777631 23472 external_mini_cluster.cc:1428] Setting key 51f5a063d3f71a9c4cec7e5a775c7066
Sep 25 15:04:40 dist-test-slave-nsrc krb5kdc[27550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348680, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:40.788470 27784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.772237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:54335 (local address 127.22.236.62:46507)
0925 15:04:40.772453 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:40.772459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:40.773587 (+ 1128us) server_negotiation.cc:407] Connection header received
0925 15:04:40.775375 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:40.775380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:40.775457 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:40.775541 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:40.777215 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.777994 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.779613 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.780038 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.780165 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.783418 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:40.783423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:40.783426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:40.783457 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:40.785686 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:40.786376 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:40.786381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:40.786383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:40.786498 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:40.786896 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:40.786906 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:40.786908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:40.787184 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:40.787301 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:40.787955 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 15:04:40.788106 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:04:40.789330 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.772746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:46507 (local address 127.22.236.3:54335)
0925 15:04:40.773426 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:40.773466 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:40.775089 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:40.775717 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:40.775724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:40.776334 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:40.777016 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:40.777043 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.778142 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.778146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:40.779493 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:40.779507 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.780276 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.780284 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:40.780487 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.781099 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:40.781122 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:40.783232 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:40.785801 (+ 2569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:40.785808 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:40.785847 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:40.786229 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:40.786584 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:40.786587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:40.786590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:40.786741 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:40.787296 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:40.787301 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:40.787588 (+ 287us) client_negotiation.cc:769] Sending connection context
0925 15:04:40.787802 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 15:04:40.788197 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":273,"threads_started":1}
I20260925 15:04:40.790596 28047 heartbeater.cc:347] Connected to a master server at 127.22.236.62:46507
I20260925 15:04:40.790889 28047 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:40.791610 28047 heartbeater.cc:511] Master 127.22.236.62:46507 requested a full tablet report, sending...
I20260925 15:04:40.793001 27594 ts_manager.cc:194] Registered new tserver with Master: f46d54b1aeeb4817ad111bac4f054771 (127.22.236.3:43825)
I20260925 15:04:40.793713 27594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:54335
I20260925 15:04:40.804929 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:04:40.820926 27784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.809234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60862 (local address 127.22.236.62:46507)
0925 15:04:40.809381 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:40.809385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:40.809849 (+ 464us) server_negotiation.cc:407] Connection header received
0925 15:04:40.809982 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:40.809986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:40.810050 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:40.810126 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:40.811446 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.812234 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.813485 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.813906 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.813992 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.816076 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:40.816082 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:40.816085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:40.816115 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:40.818709 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:40.819313 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:40.819316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:40.819318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:40.819436 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:40.819703 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:40.819706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:40.819708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:40.820014 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:40.820202 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:40.820502 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 15:04:40.820651 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:04:40.824681 27594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:04:40.831238 27594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:04:40.852977 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.844618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:34793 (local address 127.0.0.1:46174)
0925 15:04:40.845435 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:40.845456 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:40.845583 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:40.846505 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:40.846509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:40.846530 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:40.846884 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:40.846895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.848419 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.848424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:40.850548 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:40.850562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.851833 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.851848 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:40.852023 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.852071 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:04:40.852369 (+ 298us) client_negotiation.cc:240] Negotiation successful
0925 15:04:40.852579 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":3,"thread_start_us":570,"threads_started":1}
I20260925 15:04:40.853387 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.843785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:43825 (local address 127.0.0.1:36380)
0925 15:04:40.844494 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:40.844510 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:40.844607 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:40.845659 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:40.845663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:40.845685 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:40.846026 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:40.846038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.848019 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.848024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:40.850592 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:40.850605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.852501 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.852509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:40.852743 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.852773 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:04:40.852856 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 15:04:40.852999 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":192,"thread_start_us":491,"threads_started":1}
I20260925 15:04:40.853590 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.844131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36380 (local address 127.22.236.3:43825)
0925 15:04:40.845138 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:40.845146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:40.845173 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:04:40.845258 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:40.845265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:40.845465 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:40.846207 (+ 742us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:40.846467 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.847907 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.850767 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.852196 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.852618 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.852825 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:40.852899 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 15:04:40.853041 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":658,"threads_started":1}
I20260925 15:04:40.854003 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.842918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.22.236.2:33611)
0925 15:04:40.844249 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:40.844257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:40.844289 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:04:40.844383 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:40.844391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:40.844656 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:40.844850 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:40.846222 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.847733 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.850767 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.852196 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.852326 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.852511 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:40.853064 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 15:04:40.853229 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1156,"thread_start_us":1028,"threads_started":1}
I20260925 15:04:40.854118 28059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.842622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:33611 (local address 127.0.0.1:39538)
0925 15:04:40.843429 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:40.843465 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:40.843626 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:40.845155 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:40.845163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:40.845194 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:40.845844 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:40.846054 (+ 210us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.847872 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.847878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:40.850597 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:40.850605 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.852458 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.852465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:40.852663 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.852744 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 15:04:40.852824 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 15:04:40.852956 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":162,"thread_start_us":405,"threads_started":1}
I20260925 15:04:40.854300 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.845075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.22.236.1:34793)
0925 15:04:40.845993 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:40.845999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:40.846019 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:04:40.846099 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:40.846105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:40.846307 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:40.847027 (+ 720us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:40.847174 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.848302 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.850690 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.851669 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.851792 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.851990 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:40.853464 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0925 15:04:40.853618 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":706,"threads_started":1}
I20260925 15:04:40.857125 27982 tablet_service.cc:1467] Processing CreateTablet for tablet 037a4909da02466db0d81c17dd0536d8 (DEFAULT_TABLE table=test-table [id=9c09709f0a914469b92ad455b3623e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:40.857522 27717 tablet_service.cc:1467] Processing CreateTablet for tablet 037a4909da02466db0d81c17dd0536d8 (DEFAULT_TABLE table=test-table [id=9c09709f0a914469b92ad455b3623e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:40.857522 27850 tablet_service.cc:1467] Processing CreateTablet for tablet 037a4909da02466db0d81c17dd0536d8 (DEFAULT_TABLE table=test-table [id=9c09709f0a914469b92ad455b3623e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:40.858271 27982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037a4909da02466db0d81c17dd0536d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:40.858760 27717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037a4909da02466db0d81c17dd0536d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:40.859242 27850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037a4909da02466db0d81c17dd0536d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:40.872095 28065 tablet_bootstrap.cc:492] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771: Bootstrap starting.
I20260925 15:04:40.873979 28066 tablet_bootstrap.cc:492] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81: Bootstrap starting.
I20260925 15:04:40.875464 28067 tablet_bootstrap.cc:492] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d: Bootstrap starting.
I20260925 15:04:40.875586 28065 tablet_bootstrap.cc:654] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:40.876863 28065 log.cc:824] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:40.877444 28066 tablet_bootstrap.cc:654] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:40.878373 28067 tablet_bootstrap.cc:654] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:40.878476 28066 log.cc:824] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:40.879347 28067 log.cc:824] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:40.879776 28065 tablet_bootstrap.cc:492] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771: No bootstrap required, opened a new log
I20260925 15:04:40.880082 28065 ts_tablet_manager.cc:1397] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260925 15:04:40.881206 28066 tablet_bootstrap.cc:492] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81: No bootstrap required, opened a new log
I20260925 15:04:40.881524 28066 ts_tablet_manager.cc:1397] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:04:40.882025 28067 tablet_bootstrap.cc:492] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d: No bootstrap required, opened a new log
I20260925 15:04:40.882230 28067 ts_tablet_manager.cc:1397] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260925 15:04:40.888967 28067 raft_consensus.cc:348] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2326db6f66a4c0181140f3a9d68db81" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33611 } } peers { permanent_uuid: "f46d54b1aeeb4817ad111bac4f054771" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43825 } } peers { permanent_uuid: "e04c180a83bd449dae45fdea321b891d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34793 } }
I20260925 15:04:40.889304 28067 raft_consensus.cc:374] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:40.889407 28067 raft_consensus.cc:729] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04c180a83bd449dae45fdea321b891d, State: Initialized, Role: FOLLOWER
I20260925 15:04:40.889973 28067 consensus_queue.cc:260] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2326db6f66a4c0181140f3a9d68db81" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33611 } } peers { permanent_uuid: "f46d54b1aeeb4817ad111bac4f054771" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43825 } } peers { permanent_uuid: "e04c180a83bd449dae45fdea321b891d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34793 } }
I20260925 15:04:40.890066 28065 raft_consensus.cc:348] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2326db6f66a4c0181140f3a9d68db81" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33611 } } peers { permanent_uuid: "f46d54b1aeeb4817ad111bac4f054771" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43825 } } peers { permanent_uuid: "e04c180a83bd449dae45fdea321b891d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34793 } }
I20260925 15:04:40.890760 27782 heartbeater.cc:503] Master 127.22.236.62:46507 was elected leader, sending a full tablet report...
I20260925 15:04:40.891240 28066 raft_consensus.cc:348] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2326db6f66a4c0181140f3a9d68db81" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33611 } } peers { permanent_uuid: "f46d54b1aeeb4817ad111bac4f054771" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43825 } } peers { permanent_uuid: "e04c180a83bd449dae45fdea321b891d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34793 } }
I20260925 15:04:40.891628 28066 raft_consensus.cc:374] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:40.891650 28067 ts_tablet_manager.cc:1428] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260925 15:04:40.891707 28066 raft_consensus.cc:729] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2326db6f66a4c0181140f3a9d68db81, State: Initialized, Role: FOLLOWER
I20260925 15:04:40.892047 28065 raft_consensus.cc:374] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:40.892208 28065 raft_consensus.cc:729] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46d54b1aeeb4817ad111bac4f054771, State: Initialized, Role: FOLLOWER
I20260925 15:04:40.892354 28066 consensus_queue.cc:260] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2326db6f66a4c0181140f3a9d68db81" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33611 } } peers { permanent_uuid: "f46d54b1aeeb4817ad111bac4f054771" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43825 } } peers { permanent_uuid: "e04c180a83bd449dae45fdea321b891d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34793 } }
I20260925 15:04:40.892769 28065 consensus_queue.cc:260] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2326db6f66a4c0181140f3a9d68db81" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33611 } } peers { permanent_uuid: "f46d54b1aeeb4817ad111bac4f054771" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43825 } } peers { permanent_uuid: "e04c180a83bd449dae45fdea321b891d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34793 } }
I20260925 15:04:40.893996 28066 ts_tablet_manager.cc:1428] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260925 15:04:40.894461 27915 heartbeater.cc:503] Master 127.22.236.62:46507 was elected leader, sending a full tablet report...
I20260925 15:04:40.896766 28047 heartbeater.cc:503] Master 127.22.236.62:46507 was elected leader, sending a full tablet report...
I20260925 15:04:40.897399 28065 ts_tablet_manager.cc:1428] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
I20260925 15:04:40.935534 28072 raft_consensus.cc:482] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:04:40.935819 28072 raft_consensus.cc:504] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2326db6f66a4c0181140f3a9d68db81" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33611 } } peers { permanent_uuid: "f46d54b1aeeb4817ad111bac4f054771" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43825 } } peers { permanent_uuid: "e04c180a83bd449dae45fdea321b891d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34793 } }
I20260925 15:04:40.937433 28072 leader_election.cc:288] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46d54b1aeeb4817ad111bac4f054771 (127.22.236.3:43825), e04c180a83bd449dae45fdea321b891d (127.22.236.1:34793)
I20260925 15:04:40.943825 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.937838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:33227 (local address 127.22.236.3:43825)
0925 15:04:40.938057 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:40.938065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:40.938506 (+ 441us) server_negotiation.cc:407] Connection header received
0925 15:04:40.938733 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:40.938739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:40.938811 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:40.938872 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:40.939583 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.940556 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.942276 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.943021 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.943157 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.943228 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:40.943490 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 15:04:40.943605 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:04:40.943992 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.937785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:43825 (local address 127.22.236.2:33227)
0925 15:04:40.938354 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:40.938374 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:40.938557 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:40.939044 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:40.939048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:40.939083 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:40.939439 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:40.939450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.940679 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.940683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:40.942139 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:40.942150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.943210 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.943216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:40.943346 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.943370 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 15:04:40.943458 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 15:04:40.943594 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":22,"thread_start_us":321,"threads_started":1}
I20260925 15:04:40.944057 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.938708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:57485 (local address 127.22.236.1:34793)
0925 15:04:40.938883 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:40.938888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:40.939067 (+ 179us) server_negotiation.cc:407] Connection header received
0925 15:04:40.939279 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:40.939284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:40.939355 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:40.939446 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:40.939980 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.940812 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.942472 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.943222 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.943351 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.943419 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:40.943706 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 15:04:40.943824 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:04:40.944442 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.938452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:34793 (local address 127.22.236.2:57485)
0925 15:04:40.938977 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:40.938990 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:40.939141 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:40.939482 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:40.939486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:40.939511 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:40.939816 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:40.939824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.940940 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.940945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:40.942330 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:40.942341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.943448 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.943455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:40.943579 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.943600 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:04:40.943672 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:04:40.943789 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":21,"spinlock_wait_cycles":30592,"thread_start_us":264,"threads_started":1}
I20260925 15:04:40.945056 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037a4909da02466db0d81c17dd0536d8" candidate_uuid: "f2326db6f66a4c0181140f3a9d68db81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46d54b1aeeb4817ad111bac4f054771" is_pre_election: true
I20260925 15:04:40.945200 27737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037a4909da02466db0d81c17dd0536d8" candidate_uuid: "f2326db6f66a4c0181140f3a9d68db81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04c180a83bd449dae45fdea321b891d" is_pre_election: true
I20260925 15:04:40.945503 28002 raft_consensus.cc:2454] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2326db6f66a4c0181140f3a9d68db81 in term 0.
I20260925 15:04:40.945631 27737 raft_consensus.cc:2454] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2326db6f66a4c0181140f3a9d68db81 in term 0.
I20260925 15:04:40.946077 27804 leader_election.cc:302] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2326db6f66a4c0181140f3a9d68db81, f46d54b1aeeb4817ad111bac4f054771; no voters:
I20260925 15:04:40.946403 28072 raft_consensus.cc:2781] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:04:40.946532 28072 raft_consensus.cc:482] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:04:40.946623 28072 raft_consensus.cc:3037] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:40.948699 28072 raft_consensus.cc:504] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2326db6f66a4c0181140f3a9d68db81" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33611 } } peers { permanent_uuid: "f46d54b1aeeb4817ad111bac4f054771" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43825 } } peers { permanent_uuid: "e04c180a83bd449dae45fdea321b891d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34793 } }
I20260925 15:04:40.949200 28072 leader_election.cc:288] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [CANDIDATE]: Term 1 election: Requested vote from peers f46d54b1aeeb4817ad111bac4f054771 (127.22.236.3:43825), e04c180a83bd449dae45fdea321b891d (127.22.236.1:34793)
I20260925 15:04:40.949599 28002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037a4909da02466db0d81c17dd0536d8" candidate_uuid: "f2326db6f66a4c0181140f3a9d68db81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46d54b1aeeb4817ad111bac4f054771"
I20260925 15:04:40.949672 27737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037a4909da02466db0d81c17dd0536d8" candidate_uuid: "f2326db6f66a4c0181140f3a9d68db81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04c180a83bd449dae45fdea321b891d"
I20260925 15:04:40.949759 28002 raft_consensus.cc:3037] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:40.949833 27737 raft_consensus.cc:3037] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:40.951902 28002 raft_consensus.cc:2454] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2326db6f66a4c0181140f3a9d68db81 in term 1.
I20260925 15:04:40.951932 27737 raft_consensus.cc:2454] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2326db6f66a4c0181140f3a9d68db81 in term 1.
I20260925 15:04:40.952231 27804 leader_election.cc:302] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2326db6f66a4c0181140f3a9d68db81, f46d54b1aeeb4817ad111bac4f054771; no voters:
I20260925 15:04:40.952473 28072 raft_consensus.cc:2781] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:40.952852 28072 raft_consensus.cc:686] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [term 1 LEADER]: Becoming Leader. State: Replica: f2326db6f66a4c0181140f3a9d68db81, State: Running, Role: LEADER
I20260925 15:04:40.953289 28072 consensus_queue.cc:237] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2326db6f66a4c0181140f3a9d68db81" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33611 } } peers { permanent_uuid: "f46d54b1aeeb4817ad111bac4f054771" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43825 } } peers { permanent_uuid: "e04c180a83bd449dae45fdea321b891d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34793 } }
I20260925 15:04:40.957578 27593 catalog_manager.cc:5495] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2326db6f66a4c0181140f3a9d68db81 (127.22.236.2). New cstate: current_term: 1 leader_uuid: "f2326db6f66a4c0181140f3a9d68db81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2326db6f66a4c0181140f3a9d68db81" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46d54b1aeeb4817ad111bac4f054771" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e04c180a83bd449dae45fdea321b891d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34793 } health_report { overall_health: UNKNOWN } } }
I20260925 15:04:40.974823 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:40.970235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.22.236.2:33611)
0925 15:04:40.970391 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:40.970406 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:40.970514 (+ 108us) server_negotiation.cc:407] Connection header received
0925 15:04:40.970673 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:40.970676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:40.970747 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:40.970860 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:40.971166 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.971815 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.973075 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:40.973460 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:40.973545 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:40.973901 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:40.974369 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:40.974482 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:40.974565 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 15:04:40.974626 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:04:40.982300 28002 raft_consensus.cc:1261] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771 [term 1 FOLLOWER]: Refusing update from remote peer f2326db6f66a4c0181140f3a9d68db81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:40.982295 27737 raft_consensus.cc:1261] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d [term 1 FOLLOWER]: Refusing update from remote peer f2326db6f66a4c0181140f3a9d68db81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:40.983145 28076 consensus_queue.cc:1035] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e04c180a83bd449dae45fdea321b891d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:40.983453 28072 consensus_queue.cc:1035] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46d54b1aeeb4817ad111bac4f054771" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:40.993775 28079 mvcc.cc:205] Tried to move back new op lower bound from 7333268197294485504 to 7333268197188022272. Current Snapshot: MvccSnapshot[applied={T|T < 7333268197294485504}]
I20260925 15:04:41.008447 27593 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:41.008737 27593 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:41.012228 27593 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 135ed50a2e6748019485827df97e5598: Sending DeleteTablet for 3 replicas of tablet 037a4909da02466db0d81c17dd0536d8
I20260925 15:04:41.013200 27850 tablet_service.cc:1514] Processing DeleteTablet for tablet 037a4909da02466db0d81c17dd0536d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:41 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:39538
I20260925 15:04:41.013335 27982 tablet_service.cc:1514] Processing DeleteTablet for tablet 037a4909da02466db0d81c17dd0536d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:41 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:36380
I20260925 15:04:41.013335 27717 tablet_service.cc:1514] Processing DeleteTablet for tablet 037a4909da02466db0d81c17dd0536d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:41 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:46174
I20260925 15:04:41.015280 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 27655
I20260925 15:04:41.025784 28091 tablet_replica.cc:330] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771: stopping tablet replica
I20260925 15:04:41.025959 28090 tablet_replica.cc:330] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d: stopping tablet replica
I20260925 15:04:41.026386 28090 raft_consensus.cc:2229] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:41.026381 28091 raft_consensus.cc:2229] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:41.026895 28091 raft_consensus.cc:2258] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:41.026895 28090 raft_consensus.cc:2258] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:41.029120 28090 ts_tablet_manager.cc:1905] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:41.030361 28091 ts_tablet_manager.cc:1905] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:41.035252 28089 tablet_replica.cc:330] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81: stopping tablet replica
I20260925 15:04:41.035696 28089 raft_consensus.cc:2229] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:04:41.036479 28089 raft_consensus.cc:2258] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:41.038666 28090 ts_tablet_manager.cc:1918] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:04:41.038828 28090 log.cc:1196] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/037a4909da02466db0d81c17dd0536d8
Sep 25 15:04:41 dist-test-slave-nsrc krb5kdc[27550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:04:41.040959 28089 ts_tablet_manager.cc:1905] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:41.044770 28091 ts_tablet_manager.cc:1918] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:04:41.045010 28091 log.cc:1196] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/037a4909da02466db0d81c17dd0536d8
I20260925 15:04:41.045557 28091 ts_tablet_manager.cc:1939] T 037a4909da02466db0d81c17dd0536d8 P f46d54b1aeeb4817ad111bac4f054771: Deleting consensus metadata
I20260925 15:04:41.047319 27579 catalog_manager.cc:4841] TS f46d54b1aeeb4817ad111bac4f054771 (127.22.236.3:43825): tablet 037a4909da02466db0d81c17dd0536d8 (table test-table [id=9c09709f0a914469b92ad455b3623e5f]) successfully deleted
I20260925 15:04:41.047741 28090 ts_tablet_manager.cc:1939] T 037a4909da02466db0d81c17dd0536d8 P e04c180a83bd449dae45fdea321b891d: Deleting consensus metadata
I20260925 15:04:41.047781 28089 ts_tablet_manager.cc:1918] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:04:41.047923 28089 log.cc:1196] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/037a4909da02466db0d81c17dd0536d8
I20260925 15:04:41.048028 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:41.016268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.22.236.1:34793)
0925 15:04:41.016454 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:41.016461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:41.027965 (+ 11504us) server_negotiation.cc:407] Connection header received
0925 15:04:41.028266 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:41.028271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:41.028349 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:41.028421 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:41.030134 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:41.031121 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:41.032826 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:41.033480 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:41.033576 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:41.041093 (+ 7517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:41.041107 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:41.041123 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:41.041164 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:41.044680 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:41.045522 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:41.045531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:41.045534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:41.045674 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:41.046117 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:41.046122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:41.046125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:41.046576 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:41.046830 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:41.047367 (+ 537us) server_negotiation.cc:299] Negotiation successful
0925 15:04:41.047586 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:04:41.048696 28089 ts_tablet_manager.cc:1939] T 037a4909da02466db0d81c17dd0536d8 P f2326db6f66a4c0181140f3a9d68db81: Deleting consensus metadata
I20260925 15:04:41.049362 27777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:41.049523 27579 catalog_manager.cc:4841] TS e04c180a83bd449dae45fdea321b891d (127.22.236.1:34793): tablet 037a4909da02466db0d81c17dd0536d8 (table test-table [id=9c09709f0a914469b92ad455b3623e5f]) successfully deleted
I20260925 15:04:41.051458 27579 catalog_manager.cc:4841] TS f2326db6f66a4c0181140f3a9d68db81 (127.22.236.2:33611): tablet 037a4909da02466db0d81c17dd0536d8 (table test-table [id=9c09709f0a914469b92ad455b3623e5f]) successfully deleted
I20260925 15:04:41.157327 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 27655
I20260925 15:04:41.171304 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 27788
Sep 25 15:04:41 dist-test-slave-nsrc krb5kdc[27550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:04:41.186774 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:41.172044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.22.236.2:33611)
0925 15:04:41.172239 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:41.172244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:41.172265 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:04:41.172424 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:41.172430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:41.172510 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:41.172616 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:41.173694 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:41.174884 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:41.176217 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:41.176891 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:41.177013 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:41.181181 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:41.181196 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:41.181214 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:41.181249 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:41.184097 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:41.184996 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:41.185003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:41.185006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:41.185143 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:41.185532 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:41.185535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:41.185537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:41.185853 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:41.186077 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:41.186342 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 15:04:41.186493 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:04:41.187500 27910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:41.296830 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 27788
I20260925 15:04:41.310515 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 27920
Sep 25 15:04:41 dist-test-slave-nsrc krb5kdc[27550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:04:41.324584 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:41.311201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.22.236.3:43825)
0925 15:04:41.311362 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:41.311367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:41.311449 (+ 82us) server_negotiation.cc:407] Connection header received
0925 15:04:41.311636 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:41.311642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:41.311715 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:41.311796 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:41.313097 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:41.313802 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:41.315003 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:41.315378 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:41.315485 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:41.319571 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:41.319585 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:41.319602 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:41.319639 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:41.322131 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:41.322737 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:41.322744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:41.322746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:41.322861 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:41.323190 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:41.323194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:41.323196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:41.323523 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:41.323778 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:41.324066 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 15:04:41.324287 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:04:41.325189 28042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:41.410485 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 27920
I20260925 15:04:41.422590 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 27563
I20260925 15:04:41.423576 27624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:41.528507 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 27563
2026-09-25T15:04:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3780 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T15:04:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:41Z Disabled control of system clock
I20260925 15:04:41.585067 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:46715
--webserver_interface=127.22.236.62
--webserver_port=0
--builtin_ntp_servers=127.22.236.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:46715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:04:41.882516 28100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:41.882772 28100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:41.882889 28100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:41.889845 28100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:41.889938 28100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:41.889981 28100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:41.890058 28100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:41.890108 28100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:04:41.898859 28100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:46715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:46715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:41.900004 28100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:41.901723 28100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:41.912312 28108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:41.912333 28106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:41.912338 28105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:41.913277 28100 server_base.cc:1034] running on GCE node
I20260925 15:04:41.913980 28100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:41.915196 28100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:41.916410 28100 hybrid_clock.cc:648] HybridClock initialized: now 1790348681916373 us; error 34 us; skew 500 ppm
I20260925 15:04:41.916725 28100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:04:41.918767 28100 webserver.cc:460] Webserver started at http://127.22.236.62:36035/ using document root <none> and password file <none>
I20260925 15:04:41.919407 28100 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:41.919492 28100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:41.919770 28100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:41.922428 28100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "9c290d754c214aeabffdb20dd0181d22"
format_stamp: "Formatted at 2026-09-25 15:04:41 on dist-test-slave-nsrc"
server_key: "61db2add6fa8332207daed0dadd5fb51"
server_key_iv: "f52afae2a7c3c1672a2d00cc2c2d4217"
server_key_version: "encryptionkey@0"
I20260925 15:04:41.923149 28100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "9c290d754c214aeabffdb20dd0181d22"
format_stamp: "Formatted at 2026-09-25 15:04:41 on dist-test-slave-nsrc"
server_key: "61db2add6fa8332207daed0dadd5fb51"
server_key_iv: "f52afae2a7c3c1672a2d00cc2c2d4217"
server_key_version: "encryptionkey@0"
I20260925 15:04:41.927723 28100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 15:04:41.930860 28114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:41.932096 28100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:04:41.932277 28100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "9c290d754c214aeabffdb20dd0181d22"
format_stamp: "Formatted at 2026-09-25 15:04:41 on dist-test-slave-nsrc"
server_key: "61db2add6fa8332207daed0dadd5fb51"
server_key_iv: "f52afae2a7c3c1672a2d00cc2c2d4217"
server_key_version: "encryptionkey@0"
I20260925 15:04:41.932456 28100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:41.937734 28100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:41.938416 28100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:41.938661 28100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:41.954686 28100 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:46715
I20260925 15:04:41.954727 28165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:46715 every 8 connection(s)
I20260925 15:04:41.955880 28100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:41.959491 28166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:41.959506 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 28100
I20260925 15:04:41.959666 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:41.960460 23472 external_mini_cluster.cc:1428] Setting key 4bf100f7458219082df0c72787ffd17b
I20260925 15:04:41.968364 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:41.962410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.22.236.62:46715)
0925 15:04:41.963369 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:41.963381 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:41.963421 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:04:41.965236 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:41.965257 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:41.965656 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:41.965966 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:41.966927 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:41.966933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:41.966945 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:41.966959 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:41.967014 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:41.967029 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:41.967176 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:41.967269 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 15:04:41.967472 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":352,"threads_started":1}
I20260925 15:04:41.972124 28166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22: Bootstrap starting.
I20260925 15:04:41.974848 28166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:41.975764 28166 log.cc:824] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:41.978250 28166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22: No bootstrap required, opened a new log
I20260925 15:04:41.984866 28166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c290d754c214aeabffdb20dd0181d22" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 46715 } }
I20260925 15:04:41.985185 28166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:41.985276 28166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c290d754c214aeabffdb20dd0181d22, State: Initialized, Role: FOLLOWER
I20260925 15:04:41.985790 28166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c290d754c214aeabffdb20dd0181d22" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 46715 } }
I20260925 15:04:41.985994 28166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:41.986114 28166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:41.986244 28166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:41.988281 28166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c290d754c214aeabffdb20dd0181d22" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 46715 } }
I20260925 15:04:41.988725 28166 leader_election.cc:302] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c290d754c214aeabffdb20dd0181d22; no voters:
I20260925 15:04:41.989224 28166 leader_election.cc:288] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:41.989329 28171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:41.989934 28171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [term 1 LEADER]: Becoming Leader. State: Replica: 9c290d754c214aeabffdb20dd0181d22, State: Running, Role: LEADER
I20260925 15:04:41.990522 28171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c290d754c214aeabffdb20dd0181d22" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 46715 } }
I20260925 15:04:41.991065 28166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:41.992959 28173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c290d754c214aeabffdb20dd0181d22. Latest consensus state: current_term: 1 leader_uuid: "9c290d754c214aeabffdb20dd0181d22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c290d754c214aeabffdb20dd0181d22" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 46715 } } }
I20260925 15:04:41.993309 28173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:41.993672 28172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c290d754c214aeabffdb20dd0181d22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c290d754c214aeabffdb20dd0181d22" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 46715 } } }
I20260925 15:04:41.993999 28172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:41.994953 28177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:41.999454 28177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:42.006933 28177 catalog_manager.cc:1347] Generated new cluster ID: 3776fc060f2c41448310ebb060ae4d68
I20260925 15:04:42.007051 28177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:42.044494 28177 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:42.044937 28177 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:42.054790 28177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22: Generated new TSK 0
I20260925 15:04:42.055579 28177 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:04:42.063333 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:46715
--builtin_ntp_servers=127.22.236.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:04:42.383749 28190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:42.384049 28190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:42.384192 28190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:42.391535 28190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:42.391642 28190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:42.391784 28190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:04:42.400976 28190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:42.402262 28190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:42.404021 28190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:42.416471 28196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:42.416436 28198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:42.416900 28195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:42.417515 28190 server_base.cc:1034] running on GCE node
I20260925 15:04:42.418097 28190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:42.418983 28190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:42.420199 28190 hybrid_clock.cc:648] HybridClock initialized: now 1790348682420141 us; error 40 us; skew 500 ppm
I20260925 15:04:42.420496 28190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:04:42.422525 28190 webserver.cc:460] Webserver started at http://127.22.236.1:43355/ using document root <none> and password file <none>
I20260925 15:04:42.423166 28190 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:42.423259 28190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:42.423554 28190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:42.426268 28190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "0587bca3a0874c019565746f486af505"
format_stamp: "Formatted at 2026-09-25 15:04:42 on dist-test-slave-nsrc"
server_key: "6b3b20387825d6b621b1fe92f098342d"
server_key_iv: "ffe7172116049179f0c2c6b9611d6307"
server_key_version: "encryptionkey@0"
I20260925 15:04:42.426985 28190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "0587bca3a0874c019565746f486af505"
format_stamp: "Formatted at 2026-09-25 15:04:42 on dist-test-slave-nsrc"
server_key: "6b3b20387825d6b621b1fe92f098342d"
server_key_iv: "ffe7172116049179f0c2c6b9611d6307"
server_key_version: "encryptionkey@0"
I20260925 15:04:42.431670 28190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:04:42.434916 28204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:42.436300 28190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:42.436434 28190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "0587bca3a0874c019565746f486af505"
format_stamp: "Formatted at 2026-09-25 15:04:42 on dist-test-slave-nsrc"
server_key: "6b3b20387825d6b621b1fe92f098342d"
server_key_iv: "ffe7172116049179f0c2c6b9611d6307"
server_key_version: "encryptionkey@0"
I20260925 15:04:42.436604 28190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:42.441800 28190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:42.442523 28190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:42.442723 28190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:42.443502 28190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:42.444790 28190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:42.444873 28190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:42.444947 28190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:42.444996 28190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:42.471364 28190 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:33051
I20260925 15:04:42.471402 28316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:33051 every 8 connection(s)
I20260925 15:04:42.472589 28190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:04:42.479286 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 28190
I20260925 15:04:42.479449 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:04:42.480227 23472 external_mini_cluster.cc:1428] Setting key 41110a12520ffc9c0b9bd4b8dab21e07
I20260925 15:04:42.480387 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:42.474836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:48173 (local address 127.22.236.62:46715)
0925 15:04:42.475233 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:42.475239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:42.476457 (+ 1218us) server_negotiation.cc:407] Connection header received
0925 15:04:42.478108 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:42.478114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:42.478180 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:42.478271 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:42.479572 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:42.479578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:42.479582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:42.479602 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:42.479654 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:42.479670 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:42.479791 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:42.479973 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 15:04:42.480037 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":177,"threads_started":1}
I20260925 15:04:42.481258 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:42.475413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:46715 (local address 127.22.236.1:48173)
0925 15:04:42.476312 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:42.476353 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:42.477860 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:42.478494 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:42.478503 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:42.478593 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:42.479259 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:04:42.479262 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:42.479297 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:04:42.479380 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:42.479760 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:42.479767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:42.479772 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:04:42.480014 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 15:04:42.480246 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":236,"threads_started":1}
I20260925 15:04:42.482813 28317 heartbeater.cc:347] Connected to a master server at 127.22.236.62:46715
I20260925 15:04:42.483069 28317 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:42.483060 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:46715
--builtin_ntp_servers=127.22.236.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 15:04:42.483605 28317 heartbeater.cc:511] Master 127.22.236.62:46715 requested a full tablet report, sending...
I20260925 15:04:42.485625 28131 ts_manager.cc:194] Registered new tserver with Master: 0587bca3a0874c019565746f486af505 (127.22.236.1:33051)
W20260925 15:04:42.785246 28321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:42.785521 28321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:42.785641 28321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:42.792681 28321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:42.792790 28321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:42.792938 28321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:04:42.801745 28321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:42.803022 28321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:42.804703 28321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:42.816591 28326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:42.816641 28327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:42.816663 28329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:42.817468 28321 server_base.cc:1034] running on GCE node
I20260925 15:04:42.817981 28321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:42.818833 28321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:42.820020 28321 hybrid_clock.cc:648] HybridClock initialized: now 1790348682819970 us; error 45 us; skew 500 ppm
I20260925 15:04:42.820314 28321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:04:42.822330 28321 webserver.cc:460] Webserver started at http://127.22.236.2:33177/ using document root <none> and password file <none>
I20260925 15:04:42.822954 28321 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:42.823051 28321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:42.823354 28321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:42.826069 28321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "aeac50bfb68c4affad6e74b78b6994c4"
format_stamp: "Formatted at 2026-09-25 15:04:42 on dist-test-slave-nsrc"
server_key: "e4ab9e017c86ab8914e15acf8f274c9a"
server_key_iv: "0a386c409cb6a8e1d3cda45a3e72dc4d"
server_key_version: "encryptionkey@0"
I20260925 15:04:42.826798 28321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "aeac50bfb68c4affad6e74b78b6994c4"
format_stamp: "Formatted at 2026-09-25 15:04:42 on dist-test-slave-nsrc"
server_key: "e4ab9e017c86ab8914e15acf8f274c9a"
server_key_iv: "0a386c409cb6a8e1d3cda45a3e72dc4d"
server_key_version: "encryptionkey@0"
I20260925 15:04:42.831543 28321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 15:04:42.834751 28335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:42.836133 28321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:04:42.836346 28321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "aeac50bfb68c4affad6e74b78b6994c4"
format_stamp: "Formatted at 2026-09-25 15:04:42 on dist-test-slave-nsrc"
server_key: "e4ab9e017c86ab8914e15acf8f274c9a"
server_key_iv: "0a386c409cb6a8e1d3cda45a3e72dc4d"
server_key_version: "encryptionkey@0"
I20260925 15:04:42.836544 28321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:42.841907 28321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:42.842628 28321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:42.842866 28321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:42.843688 28321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:42.845005 28321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:42.845098 28321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:42.845201 28321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:42.845280 28321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:42.873977 28321 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:38657
I20260925 15:04:42.874043 28447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:38657 every 8 connection(s)
I20260925 15:04:42.875281 28321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:04:42.880265 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 28321
I20260925 15:04:42.880414 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:04:42.880849 23472 external_mini_cluster.cc:1428] Setting key ce81b42b56ac81a33ecb70e5a50d66b0
I20260925 15:04:42.883225 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:46715
--builtin_ntp_servers=127.22.236.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 15:04:42.883333 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:42.877748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:34379 (local address 127.22.236.62:46715)
0925 15:04:42.877923 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:42.877928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:42.879462 (+ 1534us) server_negotiation.cc:407] Connection header received
0925 15:04:42.881267 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:42.881273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:42.881339 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:42.881433 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:42.882665 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:42.882671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:42.882676 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:42.882693 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:42.882724 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:42.882734 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:42.882839 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:42.883002 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 15:04:42.883062 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:04:42.884268 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:42.878298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:46715 (local address 127.22.236.2:34379)
0925 15:04:42.879301 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:42.879350 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:42.881010 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:42.881665 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:42.881673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:42.881775 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:42.882388 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:04:42.882390 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:42.882427 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:04:42.882512 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:42.882814 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:42.882823 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:42.882831 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:04:42.883072 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 15:04:42.883351 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":275,"threads_started":1}
I20260925 15:04:42.885519 28448 heartbeater.cc:347] Connected to a master server at 127.22.236.62:46715
I20260925 15:04:42.885767 28448 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:42.886307 28448 heartbeater.cc:511] Master 127.22.236.62:46715 requested a full tablet report, sending...
I20260925 15:04:42.887676 28131 ts_manager.cc:194] Registered new tserver with Master: aeac50bfb68c4affad6e74b78b6994c4 (127.22.236.2:38657)
W20260925 15:04:43.188024 28451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:43.188318 28451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:43.188410 28451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:43.195437 28451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:43.195539 28451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:43.195644 28451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:04:43.204582 28451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:43.205782 28451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:43.207413 28451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:43.219240 28457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:43.219297 28459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:43.219276 28456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:43.220402 28451 server_base.cc:1034] running on GCE node
I20260925 15:04:43.220934 28451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:43.221757 28451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:43.222949 28451 hybrid_clock.cc:648] HybridClock initialized: now 1790348683222893 us; error 49 us; skew 500 ppm
I20260925 15:04:43.223224 28451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:04:43.225488 28451 webserver.cc:460] Webserver started at http://127.22.236.3:39211/ using document root <none> and password file <none>
I20260925 15:04:43.226117 28451 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:43.226212 28451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:43.226502 28451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:43.229139 28451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "e212754928724ec498ceabe48e6206ac"
format_stamp: "Formatted at 2026-09-25 15:04:43 on dist-test-slave-nsrc"
server_key: "09db4e7e636e686a30fedc69d4804e56"
server_key_iv: "36b5f0760d625f4dc0684abcad77b459"
server_key_version: "encryptionkey@0"
I20260925 15:04:43.229869 28451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "e212754928724ec498ceabe48e6206ac"
format_stamp: "Formatted at 2026-09-25 15:04:43 on dist-test-slave-nsrc"
server_key: "09db4e7e636e686a30fedc69d4804e56"
server_key_iv: "36b5f0760d625f4dc0684abcad77b459"
server_key_version: "encryptionkey@0"
I20260925 15:04:43.234614 28451 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:04:43.237859 28465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:43.239130 28451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:04:43.239274 28451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "e212754928724ec498ceabe48e6206ac"
format_stamp: "Formatted at 2026-09-25 15:04:43 on dist-test-slave-nsrc"
server_key: "09db4e7e636e686a30fedc69d4804e56"
server_key_iv: "36b5f0760d625f4dc0684abcad77b459"
server_key_version: "encryptionkey@0"
I20260925 15:04:43.239548 28451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:43.245047 28451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:43.246199 28451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:43.246523 28451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:43.247565 28451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:43.248893 28451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:43.248983 28451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:43.249084 28451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:43.249150 28451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:43.278095 28451 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:45219
I20260925 15:04:43.278132 28577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:45219 every 8 connection(s)
I20260925 15:04:43.279366 28451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:04:43.280391 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 28451
I20260925 15:04:43.280539 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:04:43.281050 23472 external_mini_cluster.cc:1428] Setting key 23f16454494442401ad4f643feaa647c
I20260925 15:04:43.287693 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.282094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:38271 (local address 127.22.236.62:46715)
0925 15:04:43.282283 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:43.282288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:43.283985 (+ 1697us) server_negotiation.cc:407] Connection header received
0925 15:04:43.285649 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:43.285655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:43.285717 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:43.285801 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:43.287018 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:43.287024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:43.287029 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:43.287047 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:43.287079 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:43.287091 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:43.287193 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:43.287369 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 15:04:43.287431 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:04:43.288609 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.282746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:46715 (local address 127.22.236.3:38271)
0925 15:04:43.283849 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:43.283885 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:43.285405 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:43.286028 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:43.286036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:43.286127 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:43.286703 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:04:43.286705 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:43.286743 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:04:43.286862 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:43.287172 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:43.287182 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:43.287190 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:04:43.287434 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 15:04:43.287678 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":298,"threads_started":1}
I20260925 15:04:43.289816 28578 heartbeater.cc:347] Connected to a master server at 127.22.236.62:46715
I20260925 15:04:43.290108 28578 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:43.290673 28578 heartbeater.cc:511] Master 127.22.236.62:46715 requested a full tablet report, sending...
I20260925 15:04:43.292037 28131 ts_manager.cc:194] Registered new tserver with Master: e212754928724ec498ceabe48e6206ac (127.22.236.3:45219)
I20260925 15:04:43.296169 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:04:43.302846 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.300499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57000 (local address 127.22.236.62:46715)
0925 15:04:43.300653 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:43.300657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:43.301111 (+ 454us) server_negotiation.cc:407] Connection header received
0925 15:04:43.301264 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:43.301269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:43.301322 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:43.301403 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:43.302273 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:43.302278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:43.302283 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:43.302300 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:43.302332 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:43.302343 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:43.302443 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:43.302532 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 15:04:43.302589 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:04:43.306264 28131 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:04:43.312702 28131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:04:43.340085 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.323815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:38657 (local address 127.0.0.1:40462)
0925 15:04:43.335833 (+ 12018us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:43.335850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:43.335943 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:43.337675 (+ 1732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:43.337683 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:43.337782 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:43.338677 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:04:43.338680 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:43.338714 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:04:43.338795 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:43.339244 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:43.339254 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:43.339277 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:04:43.339425 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 15:04:43.339508 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11881,"thread_start_us":10954,"threads_started":1}
I20260925 15:04:43.340608 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.323592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33051 (local address 127.0.0.1:56356)
0925 15:04:43.335228 (+ 11636us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:43.335273 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:43.335420 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:43.337715 (+ 2295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:43.337718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:43.337790 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:43.338734 (+ 944us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:04:43.338737 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:43.338748 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:04:43.338795 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:43.339743 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:43.339747 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:43.339752 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:04:43.339892 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 15:04:43.339942 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11344,"thread_start_us":11965,"threads_started":1}
I20260925 15:04:43.340950 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.324198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:45219 (local address 127.0.0.1:48854)
0925 15:04:43.337020 (+ 12822us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:43.337037 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:43.337128 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:43.338246 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:43.338250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:43.338270 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:43.339141 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:04:43.339145 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:43.339159 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:04:43.339196 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:43.340089 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:43.340094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:43.340099 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:04:43.340255 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 15:04:43.340313 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12718,"thread_start_us":12577,"threads_started":1}
I20260925 15:04:43.341167 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.324391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.22.236.2:38657)
0925 15:04:43.337097 (+ 12706us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:43.337105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:43.337139 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:04:43.337207 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:43.337215 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:43.337492 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:43.337640 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:43.339003 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:43.339012 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:43.339027 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:43.339054 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:43.339110 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:43.339132 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:43.340730 (+ 1598us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:43.340794 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 15:04:43.340875 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12559,"thread_start_us":12447,"threads_started":1}
I20260925 15:04:43.341164 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.324391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48854 (local address 127.22.236.3:45219)
0925 15:04:43.336533 (+ 12142us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:43.336547 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:43.337231 (+ 684us) server_negotiation.cc:407] Connection header received
0925 15:04:43.337302 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:43.337309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:43.337490 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:43.337640 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:43.339882 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:43.339890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:43.339900 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:43.339918 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:43.339963 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:43.339981 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:43.340176 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:43.340418 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 15:04:43.340581 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11925,"thread_start_us":12168,"threads_started":1}
I20260925 15:04:43.341547 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.323782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56356 (local address 127.22.236.1:33051)
0925 15:04:43.337317 (+ 13535us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:43.337323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:43.337346 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:04:43.337403 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:43.337408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:43.337561 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:43.338298 (+ 737us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:43.339509 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:43.339516 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:43.339527 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:43.339560 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:43.339607 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:43.339625 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:43.340985 (+ 1360us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:43.341062 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 15:04:43.341265 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13394,"thread_start_us":13288,"threads_started":1}
I20260925 15:04:43.344707 28383 tablet_service.cc:1467] Processing CreateTablet for tablet 80079c88bc7c4d31be306b82689511bf (DEFAULT_TABLE table=test-table [id=8056ba6592af409181d86ee322f46d90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:43.344751 28252 tablet_service.cc:1467] Processing CreateTablet for tablet 80079c88bc7c4d31be306b82689511bf (DEFAULT_TABLE table=test-table [id=8056ba6592af409181d86ee322f46d90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:43.344753 28513 tablet_service.cc:1467] Processing CreateTablet for tablet 80079c88bc7c4d31be306b82689511bf (DEFAULT_TABLE table=test-table [id=8056ba6592af409181d86ee322f46d90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:43.345863 28383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80079c88bc7c4d31be306b82689511bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:43.346021 28513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80079c88bc7c4d31be306b82689511bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:43.346022 28252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80079c88bc7c4d31be306b82689511bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:43.356809 28596 tablet_bootstrap.cc:492] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4: Bootstrap starting.
I20260925 15:04:43.360790 28596 tablet_bootstrap.cc:654] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:43.360910 28597 tablet_bootstrap.cc:492] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505: Bootstrap starting.
I20260925 15:04:43.361768 28596 log.cc:824] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:43.363937 28598 tablet_bootstrap.cc:492] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac: Bootstrap starting.
I20260925 15:04:43.364248 28597 tablet_bootstrap.cc:654] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:43.365470 28597 log.cc:824] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:43.365523 28596 tablet_bootstrap.cc:492] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4: No bootstrap required, opened a new log
I20260925 15:04:43.365773 28596 ts_tablet_manager.cc:1397] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260925 15:04:43.367339 28598 tablet_bootstrap.cc:654] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:43.368474 28598 log.cc:824] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:43.371417 28598 tablet_bootstrap.cc:492] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac: No bootstrap required, opened a new log
I20260925 15:04:43.371481 28597 tablet_bootstrap.cc:492] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505: No bootstrap required, opened a new log
I20260925 15:04:43.371716 28598 ts_tablet_manager.cc:1397] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 15:04:43.371865 28597 ts_tablet_manager.cc:1397] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260925 15:04:43.376132 28596 raft_consensus.cc:348] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587bca3a0874c019565746f486af505" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33051 } } peers { permanent_uuid: "aeac50bfb68c4affad6e74b78b6994c4" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38657 } } peers { permanent_uuid: "e212754928724ec498ceabe48e6206ac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45219 } }
I20260925 15:04:43.376603 28596 raft_consensus.cc:374] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:43.376734 28596 raft_consensus.cc:729] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeac50bfb68c4affad6e74b78b6994c4, State: Initialized, Role: FOLLOWER
I20260925 15:04:43.377396 28596 consensus_queue.cc:260] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587bca3a0874c019565746f486af505" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33051 } } peers { permanent_uuid: "aeac50bfb68c4affad6e74b78b6994c4" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38657 } } peers { permanent_uuid: "e212754928724ec498ceabe48e6206ac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45219 } }
I20260925 15:04:43.378481 28448 heartbeater.cc:503] Master 127.22.236.62:46715 was elected leader, sending a full tablet report...
I20260925 15:04:43.379079 28596 ts_tablet_manager.cc:1428] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4: Time spent starting tablet: real 0.013s user 0.006s sys 0.005s
I20260925 15:04:43.380383 28597 raft_consensus.cc:348] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587bca3a0874c019565746f486af505" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33051 } } peers { permanent_uuid: "aeac50bfb68c4affad6e74b78b6994c4" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38657 } } peers { permanent_uuid: "e212754928724ec498ceabe48e6206ac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45219 } }
I20260925 15:04:43.380712 28597 raft_consensus.cc:374] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:43.380807 28597 raft_consensus.cc:729] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0587bca3a0874c019565746f486af505, State: Initialized, Role: FOLLOWER
I20260925 15:04:43.381346 28597 consensus_queue.cc:260] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587bca3a0874c019565746f486af505" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33051 } } peers { permanent_uuid: "aeac50bfb68c4affad6e74b78b6994c4" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38657 } } peers { permanent_uuid: "e212754928724ec498ceabe48e6206ac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45219 } }
I20260925 15:04:43.381538 28598 raft_consensus.cc:348] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587bca3a0874c019565746f486af505" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33051 } } peers { permanent_uuid: "aeac50bfb68c4affad6e74b78b6994c4" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38657 } } peers { permanent_uuid: "e212754928724ec498ceabe48e6206ac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45219 } }
I20260925 15:04:43.382084 28598 raft_consensus.cc:374] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:43.382283 28598 raft_consensus.cc:729] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e212754928724ec498ceabe48e6206ac, State: Initialized, Role: FOLLOWER
I20260925 15:04:43.382309 28317 heartbeater.cc:503] Master 127.22.236.62:46715 was elected leader, sending a full tablet report...
I20260925 15:04:43.382929 28597 ts_tablet_manager.cc:1428] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260925 15:04:43.382861 28598 consensus_queue.cc:260] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587bca3a0874c019565746f486af505" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33051 } } peers { permanent_uuid: "aeac50bfb68c4affad6e74b78b6994c4" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38657 } } peers { permanent_uuid: "e212754928724ec498ceabe48e6206ac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45219 } }
I20260925 15:04:43.384356 28578 heartbeater.cc:503] Master 127.22.236.62:46715 was elected leader, sending a full tablet report...
I20260925 15:04:43.384474 28598 ts_tablet_manager.cc:1428] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 15:04:43.416908 28604 raft_consensus.cc:482] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:04:43.417168 28604 raft_consensus.cc:504] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587bca3a0874c019565746f486af505" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33051 } } peers { permanent_uuid: "aeac50bfb68c4affad6e74b78b6994c4" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38657 } } peers { permanent_uuid: "e212754928724ec498ceabe48e6206ac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45219 } }
I20260925 15:04:43.418871 28604 leader_election.cc:288] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0587bca3a0874c019565746f486af505 (127.22.236.1:33051), aeac50bfb68c4affad6e74b78b6994c4 (127.22.236.2:38657)
I20260925 15:04:43.421146 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.419051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33051 (local address 127.22.236.3:35751)
0925 15:04:43.419299 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:43.419317 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:43.419419 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:43.419724 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:43.419732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:43.419757 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:43.420377 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:04:43.420379 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:43.420392 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:04:43.420433 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:43.420709 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:43.420714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:43.420719 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:04:43.420860 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 15:04:43.420909 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260925 15:04:43.421154 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.419105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:35751 (local address 127.22.236.1:33051)
0925 15:04:43.419289 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:43.419295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:43.419485 (+ 190us) server_negotiation.cc:407] Connection header received
0925 15:04:43.419545 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:43.419549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:43.419594 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:43.419671 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:43.420563 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:43.420569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:43.420574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:43.420593 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:43.420619 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:43.420630 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:43.420719 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:43.420855 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 15:04:43.420909 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:04:43.421602 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.419237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:35997 (local address 127.22.236.2:38657)
0925 15:04:43.419427 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:43.419434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:43.419958 (+ 524us) server_negotiation.cc:407] Connection header received
0925 15:04:43.420069 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:43.420073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:43.420118 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:43.420229 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:43.421059 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:43.421065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:43.421070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:43.421085 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:43.421111 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:43.421122 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:43.421222 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:43.421359 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:04:43.421418 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:04:43.422200 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.419109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:38657 (local address 127.22.236.3:35997)
0925 15:04:43.419856 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:43.419869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:43.419960 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:43.420272 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:43.420276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:43.420298 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:43.420880 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:04:43.420883 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:43.420895 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:04:43.420928 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:43.421204 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:43.421209 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:43.421214 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:04:43.421784 (+ 570us) client_negotiation.cc:240] Negotiation successful
0925 15:04:43.421834 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":478,"threads_started":1}
I20260925 15:04:43.422197 28272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80079c88bc7c4d31be306b82689511bf" candidate_uuid: "e212754928724ec498ceabe48e6206ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0587bca3a0874c019565746f486af505" is_pre_election: true
I20260925 15:04:43.422744 28272 raft_consensus.cc:2454] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e212754928724ec498ceabe48e6206ac in term 0.
I20260925 15:04:43.423062 28403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80079c88bc7c4d31be306b82689511bf" candidate_uuid: "e212754928724ec498ceabe48e6206ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeac50bfb68c4affad6e74b78b6994c4" is_pre_election: true
I20260925 15:04:43.423357 28466 leader_election.cc:302] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0587bca3a0874c019565746f486af505, e212754928724ec498ceabe48e6206ac; no voters:
I20260925 15:04:43.423496 28403 raft_consensus.cc:2454] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e212754928724ec498ceabe48e6206ac in term 0.
I20260925 15:04:43.423681 28604 raft_consensus.cc:2781] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:04:43.423803 28604 raft_consensus.cc:482] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:04:43.423914 28604 raft_consensus.cc:3037] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:43.426087 28604 raft_consensus.cc:504] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587bca3a0874c019565746f486af505" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33051 } } peers { permanent_uuid: "aeac50bfb68c4affad6e74b78b6994c4" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38657 } } peers { permanent_uuid: "e212754928724ec498ceabe48e6206ac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45219 } }
I20260925 15:04:43.426640 28604 leader_election.cc:288] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [CANDIDATE]: Term 1 election: Requested vote from peers 0587bca3a0874c019565746f486af505 (127.22.236.1:33051), aeac50bfb68c4affad6e74b78b6994c4 (127.22.236.2:38657)
I20260925 15:04:43.427049 28272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80079c88bc7c4d31be306b82689511bf" candidate_uuid: "e212754928724ec498ceabe48e6206ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0587bca3a0874c019565746f486af505"
I20260925 15:04:43.427150 28403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80079c88bc7c4d31be306b82689511bf" candidate_uuid: "e212754928724ec498ceabe48e6206ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeac50bfb68c4affad6e74b78b6994c4"
I20260925 15:04:43.427220 28272 raft_consensus.cc:3037] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:43.427356 28403 raft_consensus.cc:3037] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:43.429392 28272 raft_consensus.cc:2454] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e212754928724ec498ceabe48e6206ac in term 1.
I20260925 15:04:43.429513 28403 raft_consensus.cc:2454] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e212754928724ec498ceabe48e6206ac in term 1.
I20260925 15:04:43.429715 28466 leader_election.cc:302] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0587bca3a0874c019565746f486af505, e212754928724ec498ceabe48e6206ac; no voters:
I20260925 15:04:43.429955 28604 raft_consensus.cc:2781] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:43.430333 28604 raft_consensus.cc:686] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [term 1 LEADER]: Becoming Leader. State: Replica: e212754928724ec498ceabe48e6206ac, State: Running, Role: LEADER
I20260925 15:04:43.430814 28604 consensus_queue.cc:237] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587bca3a0874c019565746f486af505" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33051 } } peers { permanent_uuid: "aeac50bfb68c4affad6e74b78b6994c4" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38657 } } peers { permanent_uuid: "e212754928724ec498ceabe48e6206ac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45219 } }
I20260925 15:04:43.435283 28129 catalog_manager.cc:5495] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac reported cstate change: term changed from 0 to 1, leader changed from <none> to e212754928724ec498ceabe48e6206ac (127.22.236.3). New cstate: current_term: 1 leader_uuid: "e212754928724ec498ceabe48e6206ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0587bca3a0874c019565746f486af505" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeac50bfb68c4affad6e74b78b6994c4" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e212754928724ec498ceabe48e6206ac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45219 } health_report { overall_health: HEALTHY } } }
I20260925 15:04:43.465804 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.464072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.22.236.3:45219)
0925 15:04:43.464257 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:43.464262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:43.464278 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:04:43.464434 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:43.464437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:43.464513 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:43.464591 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:43.465326 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:43.465331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:43.465334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:43.465348 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:43.465378 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:43.465389 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:43.465477 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:43.465530 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 15:04:43.465571 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:04:43.473615 28403 raft_consensus.cc:1261] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4 [term 1 FOLLOWER]: Refusing update from remote peer e212754928724ec498ceabe48e6206ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:43.473663 28272 raft_consensus.cc:1261] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505 [term 1 FOLLOWER]: Refusing update from remote peer e212754928724ec498ceabe48e6206ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 15:04:43.474601 28318 tablet.cc:2369] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:04:43.474502 28606 consensus_queue.cc:1035] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0587bca3a0874c019565746f486af505" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:43.475204 28604 consensus_queue.cc:1035] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeac50bfb68c4affad6e74b78b6994c4" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:43.488688 28609 mvcc.cc:205] Tried to move back new op lower bound from 7333268207498186752 to 7333268207336415232. Current Snapshot: MvccSnapshot[applied={T|T < 7333268207498186752}]
I20260925 15:04:43.501648 28129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:43.502007 28129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:43.506404 28129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c290d754c214aeabffdb20dd0181d22: Sending DeleteTablet for 3 replicas of tablet 80079c88bc7c4d31be306b82689511bf
I20260925 15:04:43.507535 28383 tablet_service.cc:1514] Processing DeleteTablet for tablet 80079c88bc7c4d31be306b82689511bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:43 UTC) from {username='slave'} at 127.0.0.1:40462
I20260925 15:04:43.507653 28382 tablet_service.cc:1514] Processing DeleteTablet for tablet 80079c88bc7c4d31be306b82689511bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:43 UTC) from {username='slave'} at 127.0.0.1:40462
I20260925 15:04:43.507930 28252 tablet_service.cc:1514] Processing DeleteTablet for tablet 80079c88bc7c4d31be306b82689511bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:43 UTC) from {username='slave'} at 127.0.0.1:56356
I20260925 15:04:43.508114 28513 tablet_service.cc:1514] Processing DeleteTablet for tablet 80079c88bc7c4d31be306b82689511bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:43 UTC) from {username='slave'} at 127.0.0.1:48854
I20260925 15:04:43.509651 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 28190
I20260925 15:04:43.523522 28621 tablet_replica.cc:330] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac: stopping tablet replica
I20260925 15:04:43.524214 28621 raft_consensus.cc:2229] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:04:43.525226 28621 raft_consensus.cc:2258] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:43.527664 28621 ts_tablet_manager.cc:1905] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:43.535167 28621 ts_tablet_manager.cc:1918] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:04:43.535482 28621 log.cc:1196] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/80079c88bc7c4d31be306b82689511bf
I20260925 15:04:43.536124 28621 ts_tablet_manager.cc:1939] T 80079c88bc7c4d31be306b82689511bf P e212754928724ec498ceabe48e6206ac: Deleting consensus metadata
I20260925 15:04:43.538160 28117 catalog_manager.cc:4841] TS e212754928724ec498ceabe48e6206ac (127.22.236.3:45219): tablet 80079c88bc7c4d31be306b82689511bf (table test-table [id=8056ba6592af409181d86ee322f46d90]) successfully deleted
I20260925 15:04:43.540529 28619 tablet_replica.cc:330] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4: stopping tablet replica
I20260925 15:04:43.541045 28619 raft_consensus.cc:2229] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:43.542464 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.512924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.22.236.1:33051)
0925 15:04:43.513113 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:43.513120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:43.538539 (+ 25419us) server_negotiation.cc:407] Connection header received
0925 15:04:43.539033 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:43.539040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:43.539174 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:43.539251 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:43.541630 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:43.541637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:43.541642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:43.541667 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:43.541711 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:43.541728 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:43.541833 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:43.542054 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 15:04:43.542122 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:04:43.543661 28312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:43.543728 28619 raft_consensus.cc:2258] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:43.546170 28619 ts_tablet_manager.cc:1905] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:43.553709 28619 ts_tablet_manager.cc:1918] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:04:43.553954 28619 log.cc:1196] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/80079c88bc7c4d31be306b82689511bf
I20260925 15:04:43.554520 28619 ts_tablet_manager.cc:1939] T 80079c88bc7c4d31be306b82689511bf P aeac50bfb68c4affad6e74b78b6994c4: Deleting consensus metadata
I20260925 15:04:43.556536 28118 catalog_manager.cc:4841] TS aeac50bfb68c4affad6e74b78b6994c4 (127.22.236.2:38657): tablet 80079c88bc7c4d31be306b82689511bf (table test-table [id=8056ba6592af409181d86ee322f46d90]) successfully deleted
W20260925 15:04:43.557192 28118 catalog_manager.cc:4808] TS aeac50bfb68c4affad6e74b78b6994c4 (127.22.236.2:38657): delete failed for tablet 80079c88bc7c4d31be306b82689511bf because the tablet was not found. No further retry: Not found: Tablet not found: 80079c88bc7c4d31be306b82689511bf
I20260925 15:04:43.640895 28620 tablet_replica.cc:330] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505: stopping tablet replica
I20260925 15:04:43.641307 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 28190
I20260925 15:04:43.641345 28620 raft_consensus.cc:2229] T 80079c88bc7c4d31be306b82689511bf P 0587bca3a0874c019565746f486af505 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 15:04:43.654533 28115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.236.1:33051 (error 108)
I20260925 15:04:43.655431 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 28321
I20260925 15:04:43.656733 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.656133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33051 (local address 127.0.0.1:56376)
0925 15:04:43.656439 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:43.656576 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260925 15:04:43.656983 28115 catalog_manager.cc:4584] TS 0587bca3a0874c019565746f486af505 (127.22.236.1:33051): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80079c88bc7c4d31be306b82689511bf: Network error: Client connection negotiation failed: client connection to 127.22.236.1:33051: connect: Connection refused (error 111)
I20260925 15:04:43.658406 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.656232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.22.236.2:38657)
0925 15:04:43.656439 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:43.656445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:43.656509 (+ 64us) server_negotiation.cc:407] Connection header received
0925 15:04:43.656700 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:43.656704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:43.656789 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:43.656889 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:43.657839 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:43.657843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:43.657846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:43.657861 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:43.657897 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:43.657908 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:43.658016 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:43.658099 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 15:04:43.658156 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 15:04:43.658978 28443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:43.710701 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.710083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33051 (local address 127.0.0.1:56386)
0925 15:04:43.710399 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:43.710509 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260925 15:04:43.711810 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.711435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33051 (local address 127.0.0.1:56394)
0925 15:04:43.711586 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:43.711673 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:04:43.756587 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 28321
I20260925 15:04:43.772473 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.771975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33051 (local address 127.0.0.1:56400)
0925 15:04:43.772201 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:43.772325 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 15:04:43.773463 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.773128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33051 (local address 127.0.0.1:56408)
0925 15:04:43.773272 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:43.773338 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:04:43.776075 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 28451
I20260925 15:04:43.778820 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.776993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48872 (local address 127.22.236.3:45219)
0925 15:04:43.777144 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:43.777148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:43.777163 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:04:43.777229 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:43.777232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:43.777281 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:43.777358 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:43.778237 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:43.778242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:43.778245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:43.778258 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:43.778288 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:43.778302 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:43.778406 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:43.778510 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 15:04:43.778554 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:04:43.779525 28573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:43.854416 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.853937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33051 (local address 127.0.0.1:56420)
0925 15:04:43.854179 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:43.854278 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 15:04:43.855440 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:43.855058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33051 (local address 127.0.0.1:56436)
0925 15:04:43.855249 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:43.855320 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 15:04:43.895550 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 28451
I20260925 15:04:43.910106 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 28100
I20260925 15:04:43.911195 28161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:44.016006 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 28100
2026-09-25T15:04:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2486 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 15:04:44.051458 23472 mini_oidc.cc:152] Starting JWKS server
I20260925 15:04:44.053901 23472 webserver.cc:460] Webserver started at https://127.0.0.1:37401/ using document root <none> and password file <none>
I20260925 15:04:44.054160 23472 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:04:44.054749 23472 webserver.cc:460] Webserver started at http://127.0.0.1:33141/ using document root <none> and password file <none>
2026-09-25T15:04:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:44Z Disabled control of system clock
I20260925 15:04:44.077818 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:33477
--webserver_interface=127.22.236.62
--webserver_port=0
--builtin_ntp_servers=127.22.236.20:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:33477
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37401/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/testchainca.cert with env {}
W20260925 15:04:44.377434 28633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:44.377729 28633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:44.377843 28633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:44.385082 28633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:44.385201 28633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:44.385242 28633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:44.385320 28633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:44.385380 28633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:04:44.385471 28633 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:04:44.385521 28633 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37401/jwks.json
I20260925 15:04:44.395414 28633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:33477
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:33477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37401/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:44.396752 28633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:44.398047 28633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:44.409036 28638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:44.409003 28639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:44.409061 28641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:44.410089 28633 server_base.cc:1034] running on GCE node
I20260925 15:04:44.410828 28633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:44.412209 28633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:44.413389 28633 hybrid_clock.cc:648] HybridClock initialized: now 1790348684413340 us; error 46 us; skew 500 ppm
I20260925 15:04:44.413697 28633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:04:44.421722 28633 webserver.cc:460] Webserver started at http://127.22.236.62:42199/ using document root <none> and password file <none>
I20260925 15:04:44.422401 28633 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:44.422516 28633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:44.422801 28633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:44.425688 28633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "d724dfc228bd481ba0da12bd1b505f6d"
format_stamp: "Formatted at 2026-09-25 15:04:44 on dist-test-slave-nsrc"
server_key: "980b006074f0327ad06ce7e2003a34b0"
server_key_iv: "4d603ce3f20bb58770af38d059c426cc"
server_key_version: "encryptionkey@0"
I20260925 15:04:44.426393 28633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "d724dfc228bd481ba0da12bd1b505f6d"
format_stamp: "Formatted at 2026-09-25 15:04:44 on dist-test-slave-nsrc"
server_key: "980b006074f0327ad06ce7e2003a34b0"
server_key_iv: "4d603ce3f20bb58770af38d059c426cc"
server_key_version: "encryptionkey@0"
I20260925 15:04:44.431152 28633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:04:44.434409 28647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:44.435727 28633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:04:44.435880 28633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "d724dfc228bd481ba0da12bd1b505f6d"
format_stamp: "Formatted at 2026-09-25 15:04:44 on dist-test-slave-nsrc"
server_key: "980b006074f0327ad06ce7e2003a34b0"
server_key_iv: "4d603ce3f20bb58770af38d059c426cc"
server_key_version: "encryptionkey@0"
I20260925 15:04:44.436074 28633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:44.452667 28633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:44.453483 28633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:44.453758 28633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:44.470384 28633 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:33477
I20260925 15:04:44.470404 28698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:33477 every 8 connection(s)
I20260925 15:04:44.471639 28633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:44.473606 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 28633
I20260925 15:04:44.473830 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:44.474561 23472 external_mini_cluster.cc:1428] Setting key b2212a4a5eda1850fa46cdc82a101e9a
I20260925 15:04:44.475891 28699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:44.487571 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:44.476464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.22.236.62:33477)
0925 15:04:44.477337 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:44.477348 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:44.477386 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:04:44.478861 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:44.478881 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:44.479331 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:44.479697 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:44.480250 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:44.481556 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:44.483444 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:44.484054 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:44.484213 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:44.485398 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:44.485406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:44.485424 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:44.485968 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:44.486043 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:44.486066 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:44.486157 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:44.486359 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 15:04:44.486656 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":289,"threads_started":1}
I20260925 15:04:44.488386 28699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d: Bootstrap starting.
I20260925 15:04:44.491011 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:04:44.491544 28699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:44.492727 28699 log.cc:824] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:44.495489 28699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d: No bootstrap required, opened a new log
I20260925 15:04:44.502524 28699 raft_consensus.cc:348] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d724dfc228bd481ba0da12bd1b505f6d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33477 } }
I20260925 15:04:44.502884 28699 raft_consensus.cc:374] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:44.502960 28699 raft_consensus.cc:729] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d724dfc228bd481ba0da12bd1b505f6d, State: Initialized, Role: FOLLOWER
I20260925 15:04:44.503443 28699 consensus_queue.cc:260] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d724dfc228bd481ba0da12bd1b505f6d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33477 } }
I20260925 15:04:44.503633 28699 raft_consensus.cc:388] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:44.503719 28699 raft_consensus.cc:482] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:44.503859 28699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:44.505904 28699 raft_consensus.cc:504] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d724dfc228bd481ba0da12bd1b505f6d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33477 } }
I20260925 15:04:44.506342 28699 leader_election.cc:302] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d724dfc228bd481ba0da12bd1b505f6d; no voters:
I20260925 15:04:44.506847 28699 leader_election.cc:288] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:44.507000 28704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:44.507884 28704 raft_consensus.cc:686] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [term 1 LEADER]: Becoming Leader. State: Replica: d724dfc228bd481ba0da12bd1b505f6d, State: Running, Role: LEADER
I20260925 15:04:44.508463 28704 consensus_queue.cc:237] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d724dfc228bd481ba0da12bd1b505f6d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33477 } }
I20260925 15:04:44.509058 28699 sys_catalog.cc:564] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:44.511148 28705 sys_catalog.cc:455] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d724dfc228bd481ba0da12bd1b505f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d724dfc228bd481ba0da12bd1b505f6d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33477 } } }
I20260925 15:04:44.511121 28706 sys_catalog.cc:455] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d724dfc228bd481ba0da12bd1b505f6d. Latest consensus state: current_term: 1 leader_uuid: "d724dfc228bd481ba0da12bd1b505f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d724dfc228bd481ba0da12bd1b505f6d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33477 } } }
I20260925 15:04:44.511838 28705 sys_catalog.cc:458] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:44.511838 28706 sys_catalog.cc:458] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:44.514161 28710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:44.518383 28710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:44.526166 28710 catalog_manager.cc:1347] Generated new cluster ID: 209a065f747b46838bd9acddc00e4c9a
I20260925 15:04:44.526284 28710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:44.552639 28710 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:44.553944 28710 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:44.562808 28710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d724dfc228bd481ba0da12bd1b505f6d: Generated new TSK 0
I20260925 15:04:44.563670 28710 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 15:04:44.614137 28729 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 15:04:44.614375 28729 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:04:44.615247 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:44.587592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.22.236.62:33477)
0925 15:04:44.587806 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:44.587814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:44.588376 (+ 562us) server_negotiation.cc:407] Connection header received
0925 15:04:44.588700 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:44.588704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:44.588776 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:44.588896 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:04:44.589418 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:44.590151 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:44.591511 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:44.591960 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:44.592069 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:44.592607 (+ 538us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:04:44.614554 (+ 21947us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 15:04:44.614917 (+ 363us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45484: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 15:04:44.615381 28702 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45484: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 15:04:44.616763 28728 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:44.587499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33477 (local address 127.0.0.1:45484)
0925 15:04:44.588277 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:44.588295 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:44.588461 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:44.588942 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:44.588947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:44.588959 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:04:44.589274 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:44.589283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:44.590285 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:44.590291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:44.591370 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:44.591382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:44.592133 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:44.592163 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:44.592296 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:44.592357 (+ 61us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:04:44.614856 (+ 22499us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 15:04:44.615226 (+ 370us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.236.62:33477: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":468,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.22.236.62:33477: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 15:04:44.618402 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 28633
I20260925 15:04:44.619541 28694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:44.712316 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 28633
2026-09-25T15:04:44Z chronyd exiting
I20260925 15:04:44.758450 23472 test_util.cc:175] -----------------------------------------------
I20260925 15:04:44.758594 23472 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348645297121-23472-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (711 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 15:04:44.762187 23472 mini_oidc.cc:152] Starting JWKS server
I20260925 15:04:44.763922 23472 webserver.cc:460] Webserver started at https://127.0.0.1:36519/ using document root <none> and password file <none>
I20260925 15:04:44.764132 23472 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:04:44.764798 23472 webserver.cc:460] Webserver started at http://127.0.0.1:37271/ using document root <none> and password file <none>
2026-09-25T15:04:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:44Z Disabled control of system clock
I20260925 15:04:44.786257 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:45535
--webserver_interface=127.22.236.62
--webserver_port=0
--builtin_ntp_servers=127.22.236.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:45535
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36519/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/testchainca.cert with env {}
W20260925 15:04:45.084322 28736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:45.084602 28736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:45.084791 28736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:45.093462 28736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:45.093583 28736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:45.093648 28736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:45.093724 28736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:45.093783 28736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:04:45.093869 28736 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:04:45.093920 28736 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36519/jwks.json
I20260925 15:04:45.103754 28736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:45535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:45535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36519/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:45.105062 28736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:45.106287 28736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:45.117065 28741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:45.117111 28742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:45.117131 28744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:45.117941 28736 server_base.cc:1034] running on GCE node
I20260925 15:04:45.118659 28736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:45.119935 28736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:45.121117 28736 hybrid_clock.cc:648] HybridClock initialized: now 1790348685121054 us; error 61 us; skew 500 ppm
I20260925 15:04:45.121474 28736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:04:45.123489 28736 webserver.cc:460] Webserver started at http://127.22.236.62:45731/ using document root <none> and password file <none>
I20260925 15:04:45.124114 28736 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:45.124243 28736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:45.124524 28736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:45.127470 28736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "884e93ed3ead4714b1ea3f054e201c8e"
format_stamp: "Formatted at 2026-09-25 15:04:45 on dist-test-slave-nsrc"
server_key: "6950a22334d2dc74afd70b16fad8788c"
server_key_iv: "4335631a91d94683911cd72442f74f29"
server_key_version: "encryptionkey@0"
I20260925 15:04:45.128209 28736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "884e93ed3ead4714b1ea3f054e201c8e"
format_stamp: "Formatted at 2026-09-25 15:04:45 on dist-test-slave-nsrc"
server_key: "6950a22334d2dc74afd70b16fad8788c"
server_key_iv: "4335631a91d94683911cd72442f74f29"
server_key_version: "encryptionkey@0"
I20260925 15:04:45.132853 28736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:04:45.135967 28750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:45.137233 28736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:04:45.137377 28736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "884e93ed3ead4714b1ea3f054e201c8e"
format_stamp: "Formatted at 2026-09-25 15:04:45 on dist-test-slave-nsrc"
server_key: "6950a22334d2dc74afd70b16fad8788c"
server_key_iv: "4335631a91d94683911cd72442f74f29"
server_key_version: "encryptionkey@0"
I20260925 15:04:45.137557 28736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:45.154870 28736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:45.155658 28736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:45.155941 28736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:45.171773 28736 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:45535
I20260925 15:04:45.171792 28801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:45535 every 8 connection(s)
I20260925 15:04:45.173038 28736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:45.176899 28802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:45.179697 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 28736
I20260925 15:04:45.179883 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:45.180665 23472 external_mini_cluster.cc:1428] Setting key 437a88091ef8f65e85fd213cd0f252a6
I20260925 15:04:45.188799 28802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e: Bootstrap starting.
I20260925 15:04:45.192209 28802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:45.193284 28802 log.cc:824] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:45.193670 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:45.182497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.22.236.62:45535)
0925 15:04:45.183494 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:45.183505 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:45.183547 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:04:45.185169 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:45.185193 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:45.185673 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:45.186007 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:45.186532 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:45.187745 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:45.189464 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:45.190110 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:45.190242 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:45.191449 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:45.191457 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:45.191470 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:45.191971 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:45.192052 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:45.192071 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:45.192190 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:45.192447 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 15:04:45.192753 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":398,"threads_started":1}
I20260925 15:04:45.196317 28802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e: No bootstrap required, opened a new log
I20260925 15:04:45.197568 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:04:45.204530 28802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884e93ed3ead4714b1ea3f054e201c8e" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45535 } }
I20260925 15:04:45.204910 28802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:45.204993 28802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884e93ed3ead4714b1ea3f054e201c8e, State: Initialized, Role: FOLLOWER
I20260925 15:04:45.205487 28802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884e93ed3ead4714b1ea3f054e201c8e" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45535 } }
I20260925 15:04:45.205685 28802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:45.205775 28802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:45.205899 28802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:45.207875 28802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884e93ed3ead4714b1ea3f054e201c8e" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45535 } }
I20260925 15:04:45.208338 28802 leader_election.cc:302] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 884e93ed3ead4714b1ea3f054e201c8e; no voters:
I20260925 15:04:45.208798 28802 leader_election.cc:288] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:45.208935 28807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:45.209587 28807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [term 1 LEADER]: Becoming Leader. State: Replica: 884e93ed3ead4714b1ea3f054e201c8e, State: Running, Role: LEADER
I20260925 15:04:45.210038 28807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884e93ed3ead4714b1ea3f054e201c8e" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45535 } }
I20260925 15:04:45.210620 28802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:45.212543 28808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "884e93ed3ead4714b1ea3f054e201c8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884e93ed3ead4714b1ea3f054e201c8e" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45535 } } }
I20260925 15:04:45.212914 28808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:45.212935 28809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 884e93ed3ead4714b1ea3f054e201c8e. Latest consensus state: current_term: 1 leader_uuid: "884e93ed3ead4714b1ea3f054e201c8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884e93ed3ead4714b1ea3f054e201c8e" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45535 } } }
I20260925 15:04:45.213799 28809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:45.214558 28814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:45.219245 28814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:45.227731 28814 catalog_manager.cc:1347] Generated new cluster ID: 898183152ed64689be8ea92377fdd9d5
I20260925 15:04:45.227839 28814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:45.244633 28814 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:45.245843 28814 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:45.253752 28814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 884e93ed3ead4714b1ea3f054e201c8e: Generated new TSK 0
I20260925 15:04:45.254736 28814 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 15:04:45.293315 28832 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 15:04:45.293460 28832 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:04:45.294435 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:45.274445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.22.236.62:45535)
0925 15:04:45.274650 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:45.274657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:45.275385 (+ 728us) server_negotiation.cc:407] Connection header received
0925 15:04:45.275662 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:45.275667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:45.275737 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:45.275929 (+ 192us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:04:45.276506 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:45.277254 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:45.279169 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:45.279640 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:45.279735 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:45.280305 (+ 570us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:04:45.293898 (+ 13593us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 15:04:45.294222 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44740: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 15:04:45.294566 28805 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44740: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 15:04:45.294864 28831 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:45.274323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45535 (local address 127.0.0.1:44740)
0925 15:04:45.275277 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:45.275296 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:45.275442 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:45.275961 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:45.275966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:45.275979 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:04:45.276333 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:45.276345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:45.277381 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:45.277388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:45.279020 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:45.279035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:45.279794 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:45.279801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:45.279996 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:45.280044 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:04:45.294154 (+ 14110us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 15:04:45.294461 (+ 307us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.236.62:45535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":664,"threads_started":1}
I20260925 15:04:45.296823 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 28736
I20260925 15:04:45.297950 28797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:45.387923 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 28736
2026-09-25T15:04:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (676 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 15:04:45.439077 23472 mini_oidc.cc:152] Starting JWKS server
I20260925 15:04:45.440943 23472 webserver.cc:460] Webserver started at https://127.0.0.1:46325/ using document root <none> and password file <none>
I20260925 15:04:45.441166 23472 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:04:45.441741 23472 webserver.cc:460] Webserver started at http://127.0.0.1:34167/ using document root <none> and password file <none>
2026-09-25T15:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:45Z Disabled control of system clock
I20260925 15:04:45.464570 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:43991
--webserver_interface=127.22.236.62
--webserver_port=0
--builtin_ntp_servers=127.22.236.20:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:43991
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46325/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 15:04:45.767987 28839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:45.768285 28839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:45.768379 28839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:45.775305 28839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:45.775396 28839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:45.775431 28839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:45.775485 28839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:45.775527 28839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:04:45.775583 28839 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:04:45.775619 28839 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46325/jwks.json
I20260925 15:04:45.784307 28839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:43991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:43991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46325/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:45.785462 28839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:45.787130 28839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:45.797657 28847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:45.797688 28845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:45.797711 28844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:45.798239 28839 server_base.cc:1034] running on GCE node
I20260925 15:04:45.798962 28839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:45.800253 28839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:45.801436 28839 hybrid_clock.cc:648] HybridClock initialized: now 1790348685801385 us; error 47 us; skew 500 ppm
I20260925 15:04:45.801726 28839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:04:45.803745 28839 webserver.cc:460] Webserver started at http://127.22.236.62:37369/ using document root <none> and password file <none>
I20260925 15:04:45.804414 28839 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:45.804497 28839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:45.804759 28839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:45.807502 28839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "3a121dcc6bf24eadbf3bf80365217915"
format_stamp: "Formatted at 2026-09-25 15:04:45 on dist-test-slave-nsrc"
server_key: "e3d7427a97ff7424e355badb024a64d2"
server_key_iv: "c0ac611324cb25b74c3961f87169b1a9"
server_key_version: "encryptionkey@0"
I20260925 15:04:45.808282 28839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "3a121dcc6bf24eadbf3bf80365217915"
format_stamp: "Formatted at 2026-09-25 15:04:45 on dist-test-slave-nsrc"
server_key: "e3d7427a97ff7424e355badb024a64d2"
server_key_iv: "c0ac611324cb25b74c3961f87169b1a9"
server_key_version: "encryptionkey@0"
I20260925 15:04:45.812942 28839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:04:45.815951 28853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:45.817453 28839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:04:45.817628 28839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "3a121dcc6bf24eadbf3bf80365217915"
format_stamp: "Formatted at 2026-09-25 15:04:45 on dist-test-slave-nsrc"
server_key: "e3d7427a97ff7424e355badb024a64d2"
server_key_iv: "c0ac611324cb25b74c3961f87169b1a9"
server_key_version: "encryptionkey@0"
I20260925 15:04:45.817855 28839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:45.831988 28839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:45.832801 28839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:45.833066 28839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:45.849468 28839 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:43991
I20260925 15:04:45.849488 28904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:43991 every 8 connection(s)
I20260925 15:04:45.850732 28839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:45.854131 28905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:45.859555 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 28839
I20260925 15:04:45.859776 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:45.860575 23472 external_mini_cluster.cc:1428] Setting key c9fd6850bdd55e0ec97f90f128604ef8
I20260925 15:04:45.864722 28905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915: Bootstrap starting.
I20260925 15:04:45.867604 28905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:45.868587 28905 log.cc:824] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:45.871484 28905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915: No bootstrap required, opened a new log
I20260925 15:04:45.873805 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:45.862444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.22.236.62:43991)
0925 15:04:45.863352 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:45.863362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:45.863404 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:04:45.865120 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:45.865142 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:45.865589 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:45.866017 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:45.866541 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:45.867867 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:45.869254 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:45.870053 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:45.870225 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:45.871229 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:45.871239 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:45.871257 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:45.871925 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:45.872015 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:45.872043 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:45.872173 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:45.872336 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 15:04:45.872634 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":298,"threads_started":1}
I20260925 15:04:45.877410 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:04:45.879712 28905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a121dcc6bf24eadbf3bf80365217915" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43991 } }
I20260925 15:04:45.880036 28905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:45.880128 28905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a121dcc6bf24eadbf3bf80365217915, State: Initialized, Role: FOLLOWER
I20260925 15:04:45.880635 28905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a121dcc6bf24eadbf3bf80365217915" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43991 } }
I20260925 15:04:45.880828 28905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:45.880949 28905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:45.881091 28905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:45.883062 28905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a121dcc6bf24eadbf3bf80365217915" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43991 } }
I20260925 15:04:45.883548 28905 leader_election.cc:302] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a121dcc6bf24eadbf3bf80365217915; no voters:
I20260925 15:04:45.884125 28905 leader_election.cc:288] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:45.884228 28910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:45.884874 28910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [term 1 LEADER]: Becoming Leader. State: Replica: 3a121dcc6bf24eadbf3bf80365217915, State: Running, Role: LEADER
I20260925 15:04:45.885324 28910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a121dcc6bf24eadbf3bf80365217915" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43991 } }
I20260925 15:04:45.885969 28905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:45.887768 28912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a121dcc6bf24eadbf3bf80365217915. Latest consensus state: current_term: 1 leader_uuid: "3a121dcc6bf24eadbf3bf80365217915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a121dcc6bf24eadbf3bf80365217915" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43991 } } }
I20260925 15:04:45.888172 28912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:45.889010 28911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a121dcc6bf24eadbf3bf80365217915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a121dcc6bf24eadbf3bf80365217915" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43991 } } }
I20260925 15:04:45.889336 28911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:45.890281 28919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:45.894779 28919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:45.903389 28919 catalog_manager.cc:1347] Generated new cluster ID: dc83f1c541854afba6e4c561a0b11863
I20260925 15:04:45.903496 28919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:45.925923 28919 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:45.927146 28919 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:45.934029 28919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a121dcc6bf24eadbf3bf80365217915: Generated new TSK 0
I20260925 15:04:45.934957 28919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:04:45.958391 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:45.950326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53268 (local address 127.22.236.62:43991)
0925 15:04:45.950518 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:45.950525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:45.950965 (+ 440us) server_negotiation.cc:407] Connection header received
0925 15:04:45.951171 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:45.951175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:45.951247 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:45.951373 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:04:45.951910 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:45.952945 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:45.954875 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:45.955299 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:45.955398 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:45.955957 (+ 559us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:04:45.957772 (+ 1815us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46325/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 15:04:45.958065 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53268: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46325/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 15:04:45.958566 28934 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:45.950300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43991 (local address 127.0.0.1:53268)
0925 15:04:45.950866 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:45.950884 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:45.951025 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:45.951405 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:45.951411 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:45.951423 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:04:45.951736 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:45.951747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:45.953076 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:45.953083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:45.954731 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:45.954746 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:45.955484 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:45.955491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:45.955681 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:45.955726 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:04:45.957956 (+ 2230us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46325/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 15:04:45.958176 (+ 220us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.236.62:43991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46325/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":293,"threads_started":1}
W20260925 15:04:45.958889 28908 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53268: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46325/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260925 15:04:45.959512 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 28839
I20260925 15:04:45.960603 28900 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 15:04:45.970603 28935 webserver.cc:533] Webserver: sslize failed:
E20260925 15:04:45.970688 28935 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:04:46.047861 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 28839
2026-09-25T15:04:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (659 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T15:04:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:46Z Disabled control of system clock
I20260925 15:04:46.120718 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:35887
--webserver_interface=127.22.236.62
--webserver_port=0
--builtin_ntp_servers=127.22.236.20:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:35887
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/testchainca.cert with env {}
W20260925 15:04:46.429049 28940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:46.429348 28940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:46.429473 28940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:46.436703 28940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:46.436811 28940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:46.436862 28940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:46.436939 28940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:46.436991 28940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:04:46.437074 28940 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:04:46.437124 28940 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 15:04:46.446954 28940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:35887
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:35887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:46.448236 28940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:46.449460 28940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:46.460222 28948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:46.460246 28946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:46.460791 28945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:46.461284 28940 server_base.cc:1034] running on GCE node
I20260925 15:04:46.462037 28940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:46.463346 28940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:46.464522 28940 hybrid_clock.cc:648] HybridClock initialized: now 1790348686464482 us; error 33 us; skew 500 ppm
I20260925 15:04:46.464838 28940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:04:46.466892 28940 webserver.cc:460] Webserver started at http://127.22.236.62:35455/ using document root <none> and password file <none>
I20260925 15:04:46.467535 28940 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:46.467626 28940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:46.467901 28940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:46.470567 28940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "1647a8136ec54917912f577a7b63f019"
format_stamp: "Formatted at 2026-09-25 15:04:46 on dist-test-slave-nsrc"
server_key: "d64bb03e752895f76d9d947a59e21575"
server_key_iv: "833a97904fd0afc3b7f05c47aff561c0"
server_key_version: "encryptionkey@0"
I20260925 15:04:46.471308 28940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "1647a8136ec54917912f577a7b63f019"
format_stamp: "Formatted at 2026-09-25 15:04:46 on dist-test-slave-nsrc"
server_key: "d64bb03e752895f76d9d947a59e21575"
server_key_iv: "833a97904fd0afc3b7f05c47aff561c0"
server_key_version: "encryptionkey@0"
I20260925 15:04:46.476100 28940 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260925 15:04:46.479475 28954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:46.480876 28940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:46.481037 28940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "1647a8136ec54917912f577a7b63f019"
format_stamp: "Formatted at 2026-09-25 15:04:46 on dist-test-slave-nsrc"
server_key: "d64bb03e752895f76d9d947a59e21575"
server_key_iv: "833a97904fd0afc3b7f05c47aff561c0"
server_key_version: "encryptionkey@0"
I20260925 15:04:46.481243 28940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:46.505179 28940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:46.506012 28940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:46.506271 28940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:46.522917 28940 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:35887
I20260925 15:04:46.522938 29005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:35887 every 8 connection(s)
I20260925 15:04:46.524199 28940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:46.527208 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 28940
I20260925 15:04:46.527398 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:46.528055 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:46.528198 23472 external_mini_cluster.cc:1428] Setting key fc619a145f02bfdd47b7be5073c83f5f
I20260925 15:04:46.541115 29006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019: Bootstrap starting.
I20260925 15:04:46.541641 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:46.529979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.22.236.62:35887)
0925 15:04:46.530877 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:46.530888 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:46.530931 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:04:46.532738 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:46.532767 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:46.533385 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:46.533821 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:46.534548 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:46.536266 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:46.537514 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:46.538176 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:46.538356 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:46.539422 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:46.539429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:46.539442 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:04:46.539952 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:46.540023 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:04:46.540043 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:46.540159 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:46.540391 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 15:04:46.540687 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":305,"threads_started":1}
I20260925 15:04:46.544193 29006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:46.545132 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:04:46.545310 29006 log.cc:824] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:46.547917 29006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019: No bootstrap required, opened a new log
I20260925 15:04:46.555284 29006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1647a8136ec54917912f577a7b63f019" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35887 } }
I20260925 15:04:46.555627 29006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:46.555698 29006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1647a8136ec54917912f577a7b63f019, State: Initialized, Role: FOLLOWER
I20260925 15:04:46.556187 29006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1647a8136ec54917912f577a7b63f019" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35887 } }
I20260925 15:04:46.556384 29006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:46.556478 29006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:46.556588 29006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:46.558656 29006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1647a8136ec54917912f577a7b63f019" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35887 } }
I20260925 15:04:46.559152 29006 leader_election.cc:302] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1647a8136ec54917912f577a7b63f019; no voters:
I20260925 15:04:46.559715 29006 leader_election.cc:288] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:46.559818 29011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:46.560433 29011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [term 1 LEADER]: Becoming Leader. State: Replica: 1647a8136ec54917912f577a7b63f019, State: Running, Role: LEADER
I20260925 15:04:46.560961 29011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1647a8136ec54917912f577a7b63f019" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35887 } }
I20260925 15:04:46.561671 29006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:46.563402 29013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1647a8136ec54917912f577a7b63f019. Latest consensus state: current_term: 1 leader_uuid: "1647a8136ec54917912f577a7b63f019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1647a8136ec54917912f577a7b63f019" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35887 } } }
I20260925 15:04:46.563717 29013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:46.564589 29012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1647a8136ec54917912f577a7b63f019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1647a8136ec54917912f577a7b63f019" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35887 } } }
I20260925 15:04:46.565001 29012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:46.569273 29016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:46.573478 29016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:46.581027 29016 catalog_manager.cc:1347] Generated new cluster ID: 954c7a77cf674f24ba3e4f5140eeb929
I20260925 15:04:46.581149 29016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:46.609304 29016 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:46.611138 29016 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:46.620066 29016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1647a8136ec54917912f577a7b63f019: Generated new TSK 0
I20260925 15:04:46.621096 29016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:04:46.741948 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:46.645171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.22.236.62:35887)
0925 15:04:46.645361 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:46.645366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:46.645799 (+ 433us) server_negotiation.cc:407] Connection header received
0925 15:04:46.646007 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:46.646013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:46.646100 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:46.646270 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:04:46.646646 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:46.647736 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:46.649209 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:46.649933 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:46.650069 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:46.650535 (+ 466us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:04:46.741248 (+ 90713us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 15:04:46.741654 (+ 406us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39948: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 15:04:46.742076 29009 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39948: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 15:04:46.742409 29035 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:46.645115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:35887 (local address 127.0.0.1:39948)
0925 15:04:46.645701 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:46.645716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:46.645836 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:46.646265 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:46.646268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:46.646277 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:04:46.646509 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:46.646516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:46.647937 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:46.647941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:46.649074 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:46.649086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:46.650088 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:46.650092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:46.650217 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:46.650251 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:04:46.741537 (+ 91286us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 15:04:46.741793 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.236.62:35887: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":293,"threads_started":1}
I20260925 15:04:46.743955 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 28940
I20260925 15:04:46.744928 29001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:46.844894 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 28940
2026-09-25T15:04:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (776 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (381 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (380 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:04:47 dist-test-slave-nsrc krb5kdc[29050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:04:47 dist-test-slave-nsrc krb5kdc[29050](info): set up 2 sockets
Sep 25 15:04:47 dist-test-slave-nsrc krb5kdc[29050](info): commencing operation
krb5kdc: starting...
W20260925 15:04:48.592723 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:04:48 dist-test-slave-nsrc krb5kdc[29050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:04:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:04:48.769330 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:36777
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:36777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:49.070078 29063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:49.070334 29063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:49.070457 29063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:49.077455 29063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:49.077556 29063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:49.077632 29063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:49.077700 29063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:49.077759 29063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:04:49.088443 29063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:36777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:36777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:49.090098 29063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:49.091650 29063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:49.103065 29068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:49.103108 29071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:49.103130 29069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:49.103799 29063 server_base.cc:1034] running on GCE node
I20260925 15:04:49.104549 29063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:49.105787 29063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:49.106984 29063 hybrid_clock.cc:648] HybridClock initialized: now 1790348689106929 us; error 45 us; skew 500 ppm
I20260925 15:04:49.107273 29063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:49 dist-test-slave-nsrc krb5kdc[29050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348689, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:49.110077 29063 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:04:49.111385 29063 webserver.cc:460] Webserver started at http://127.22.236.62:36567/ using document root <none> and password file <none>
I20260925 15:04:49.112001 29063 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:49.112109 29063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:49.112397 29063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:49.115106 29063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "1df4e1dc78274a438bd706ca9241e6fb"
format_stamp: "Formatted at 2026-09-25 15:04:49 on dist-test-slave-nsrc"
server_key: "51c282804a0c5e59951ab8093bea3104"
server_key_iv: "561838805ddfc6f710d3dfd2bc7bf2e5"
server_key_version: "encryptionkey@0"
I20260925 15:04:49.115803 29063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "1df4e1dc78274a438bd706ca9241e6fb"
format_stamp: "Formatted at 2026-09-25 15:04:49 on dist-test-slave-nsrc"
server_key: "51c282804a0c5e59951ab8093bea3104"
server_key_iv: "561838805ddfc6f710d3dfd2bc7bf2e5"
server_key_version: "encryptionkey@0"
I20260925 15:04:49.120432 29063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 15:04:49.123664 29078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:49.124867 29063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:49.125003 29063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "1df4e1dc78274a438bd706ca9241e6fb"
format_stamp: "Formatted at 2026-09-25 15:04:49 on dist-test-slave-nsrc"
server_key: "51c282804a0c5e59951ab8093bea3104"
server_key_iv: "561838805ddfc6f710d3dfd2bc7bf2e5"
server_key_version: "encryptionkey@0"
I20260925 15:04:49.125157 29063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:49.157161 29063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:49.161199 29063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:49.161448 29063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:49.177616 29063 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:36777
I20260925 15:04:49.177645 29129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:36777 every 8 connection(s)
I20260925 15:04:49.178912 29063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:49.182475 29130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:49.185842 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 29063
I20260925 15:04:49.186038 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:49.186641 23472 external_mini_cluster.cc:1428] Setting key 7be8a8aa60267473bf30922311c01b2e
I20260925 15:04:49.195123 29130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb: Bootstrap starting.
I20260925 15:04:49.198488 29130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:49.199419 29130 log.cc:824] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:04:49 dist-test-slave-nsrc krb5kdc[29050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:49.202226 29130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb: No bootstrap required, opened a new log
I20260925 15:04:49.208590 29133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:49.188482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.22.236.62:36777)
0925 15:04:49.189247 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:49.189262 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:49.189318 (+ 56us) server_negotiation.cc:407] Connection header received
0925 15:04:49.191066 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:49.191096 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:49.191648 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:49.192090 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:49.193399 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:49.195277 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:49.196523 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:49.197517 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:49.197688 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:49.201474 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:49.201484 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:49.201510 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:49.201554 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:49.204664 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:49.205324 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:49.205331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:49.205346 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:49.205502 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:49.205859 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:49.205862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:49.205864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:49.206394 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:49.206668 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:49.207116 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 15:04:49.207525 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":330,"threads_started":1}
I20260925 15:04:49.209694 29130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df4e1dc78274a438bd706ca9241e6fb" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36777 } }
I20260925 15:04:49.210027 29130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:49.210124 29130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df4e1dc78274a438bd706ca9241e6fb, State: Initialized, Role: FOLLOWER
I20260925 15:04:49.210659 29130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df4e1dc78274a438bd706ca9241e6fb" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36777 } }
I20260925 15:04:49.210881 29130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:49.211007 29130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:49.211174 29130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:49.213209 29130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df4e1dc78274a438bd706ca9241e6fb" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36777 } }
I20260925 15:04:49.213642 29130 leader_election.cc:302] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1df4e1dc78274a438bd706ca9241e6fb; no voters:
I20260925 15:04:49.214092 29130 leader_election.cc:288] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:49.214205 29135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:49.215193 29135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [term 1 LEADER]: Becoming Leader. State: Replica: 1df4e1dc78274a438bd706ca9241e6fb, State: Running, Role: LEADER
I20260925 15:04:49.215695 29135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df4e1dc78274a438bd706ca9241e6fb" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36777 } }
I20260925 15:04:49.216074 29130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:49.218170 29136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1df4e1dc78274a438bd706ca9241e6fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df4e1dc78274a438bd706ca9241e6fb" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36777 } } }
I20260925 15:04:49.218569 29136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:49.219976 29137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1df4e1dc78274a438bd706ca9241e6fb. Latest consensus state: current_term: 1 leader_uuid: "1df4e1dc78274a438bd706ca9241e6fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df4e1dc78274a438bd706ca9241e6fb" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36777 } } }
I20260925 15:04:49.220275 29137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:49.220884 29144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:49.224745 29144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:49.232898 29144 catalog_manager.cc:1347] Generated new cluster ID: 9bf5ea4691454d20a57eaeb1359f66b7
I20260925 15:04:49.232990 29144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:49.259519 29144 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:49.260986 29144 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:49.268447 29144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1df4e1dc78274a438bd706ca9241e6fb: Generated new TSK 0
I20260925 15:04:49.269462 29144 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:04:49.323014 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36777
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:49.631309 29156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:49.631583 29156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:49.631709 29156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:49.638963 29156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:49.639078 29156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:49.639230 29156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:04:49.648813 29156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:49.650053 29156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:49.651319 29156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:49.663470 29161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:49.663436 29162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:49.663509 29164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:49.664407 29156 server_base.cc:1034] running on GCE node
I20260925 15:04:49.665033 29156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:49.665894 29156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:49.667096 29156 hybrid_clock.cc:648] HybridClock initialized: now 1790348689667047 us; error 39 us; skew 500 ppm
I20260925 15:04:49.667400 29156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:49 dist-test-slave-nsrc krb5kdc[29050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348689, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:49.677199 29156 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:04:49.678628 29156 webserver.cc:460] Webserver started at http://127.22.236.1:38249/ using document root <none> and password file <none>
I20260925 15:04:49.679291 29156 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:49.679411 29156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:49.679688 29156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:49.682637 29156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "156532fd18554a2e8eb60a2ab3ba111e"
format_stamp: "Formatted at 2026-09-25 15:04:49 on dist-test-slave-nsrc"
server_key: "68569619461ec370891b7784b1404e3f"
server_key_iv: "dfc96b1872ac3a1d8e72ee47b2c22541"
server_key_version: "encryptionkey@0"
I20260925 15:04:49.683384 29156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "156532fd18554a2e8eb60a2ab3ba111e"
format_stamp: "Formatted at 2026-09-25 15:04:49 on dist-test-slave-nsrc"
server_key: "68569619461ec370891b7784b1404e3f"
server_key_iv: "dfc96b1872ac3a1d8e72ee47b2c22541"
server_key_version: "encryptionkey@0"
I20260925 15:04:49.688549 29156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 15:04:49.691757 29171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:49.693046 29156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:49.693183 29156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "156532fd18554a2e8eb60a2ab3ba111e"
format_stamp: "Formatted at 2026-09-25 15:04:49 on dist-test-slave-nsrc"
server_key: "68569619461ec370891b7784b1404e3f"
server_key_iv: "dfc96b1872ac3a1d8e72ee47b2c22541"
server_key_version: "encryptionkey@0"
I20260925 15:04:49.693370 29156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:49.730204 29156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:49.734014 29156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:49.734267 29156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:49.735167 29156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:49.736443 29156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:49.736521 29156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:49.736631 29156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:49.736683 29156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:49.765168 29156 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:45371
I20260925 15:04:49.765216 29283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:45371 every 8 connection(s)
I20260925 15:04:49.766366 29156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:04:49.769789 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 29156
I20260925 15:04:49.770005 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:04:49.770802 23472 external_mini_cluster.cc:1428] Setting key 427cbc336c34e95aa3315dae9b6a6415
Sep 25 15:04:49 dist-test-slave-nsrc krb5kdc[29050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348689, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
I20260925 15:04:49.789345 29286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:49.768703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:34825 (local address 127.22.236.62:36777)
0925 15:04:49.776427 (+ 7724us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:49.776432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:49.776446 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:04:49.776490 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:49.776494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:49.776547 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:49.776640 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:49.778128 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:49.778863 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:49.780438 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:49.780838 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:49.780941 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:49.784251 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:49.784256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:49.784259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:49.784288 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:49.786563 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:49.787205 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:49.787210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:49.787213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:49.787317 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:49.787699 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:49.787704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:49.787707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:49.787952 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:49.788087 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:49.788785 (+ 698us) server_negotiation.cc:299] Negotiation successful
0925 15:04:49.789007 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7630,"thread_start_us":7559,"threads_started":1}
I20260925 15:04:49.790311 29287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:49.769198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36777 (local address 127.22.236.1:34825)
0925 15:04:49.769955 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:49.770010 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:49.771769 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:49.776891 (+ 5122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:49.776900 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:49.777392 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:49.777962 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:49.777983 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:49.779033 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:49.779037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:49.780311 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:49.780325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:49.781059 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:49.781067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:49.781283 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:49.781893 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:49.781914 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:49.784035 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:49.786688 (+ 2653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:49.786696 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:49.786714 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:49.787103 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:49.787413 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:49.787418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:49.787422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:49.787601 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:49.788078 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:49.788086 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:49.788448 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 15:04:49.788688 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 15:04:49.789188 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":299,"threads_started":1}
I20260925 15:04:49.791815 29284 heartbeater.cc:347] Connected to a master server at 127.22.236.62:36777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
I20260925 15:04:49.792124 29284 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:49.792807 29284 heartbeater.cc:511] Master 127.22.236.62:36777 requested a full tablet report, sending...
I20260925 15:04:49.794754 29095 ts_manager.cc:194] Registered new tserver with Master: 156532fd18554a2e8eb60a2ab3ba111e (127.22.236.1:45371)
I20260925 15:04:49.796186 29095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:34825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:04:49.813205 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36777
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:50.118261 29290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:50.118517 29290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:50.118647 29290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:50.125686 29290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:50.125788 29290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:50.125941 29290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:04:50.135481 29290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:50.136706 29290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:50.137954 29290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:50.150094 29295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:50.150128 29296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:50.150089 29298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:50.150440 29290 server_base.cc:1034] running on GCE node
I20260925 15:04:50.151240 29290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:50.152086 29290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:50.153292 29290 hybrid_clock.cc:648] HybridClock initialized: now 1790348690153246 us; error 38 us; skew 500 ppm
I20260925 15:04:50.153584 29290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:50.156414 29290 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:04:50.157691 29290 webserver.cc:460] Webserver started at http://127.22.236.2:35081/ using document root <none> and password file <none>
I20260925 15:04:50.158315 29290 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:50.158423 29290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:50.158694 29290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:50.161435 29290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "7f5615eab4b34418a83bac46604c8c5e"
format_stamp: "Formatted at 2026-09-25 15:04:50 on dist-test-slave-nsrc"
server_key: "8806a2ae6709b1fa5112e387aa512361"
server_key_iv: "22ad4a88d2f27d70d64bffbd06a01085"
server_key_version: "encryptionkey@0"
I20260925 15:04:50.162144 29290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "7f5615eab4b34418a83bac46604c8c5e"
format_stamp: "Formatted at 2026-09-25 15:04:50 on dist-test-slave-nsrc"
server_key: "8806a2ae6709b1fa5112e387aa512361"
server_key_iv: "22ad4a88d2f27d70d64bffbd06a01085"
server_key_version: "encryptionkey@0"
I20260925 15:04:50.166853 29290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:04:50.169936 29305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:50.171111 29290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:04:50.171245 29290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "7f5615eab4b34418a83bac46604c8c5e"
format_stamp: "Formatted at 2026-09-25 15:04:50 on dist-test-slave-nsrc"
server_key: "8806a2ae6709b1fa5112e387aa512361"
server_key_iv: "22ad4a88d2f27d70d64bffbd06a01085"
server_key_version: "encryptionkey@0"
I20260925 15:04:50.171407 29290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:50.194710 29290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:50.199332 29290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:50.199571 29290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:50.200428 29290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:50.201699 29290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:50.201766 29290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:50.201853 29290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:50.201895 29290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:50.229918 29290 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:37717
I20260925 15:04:50.229984 29417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:37717 every 8 connection(s)
I20260925 15:04:50.231079 29290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:04:50.240272 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 29290
I20260925 15:04:50.240440 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:04:50.240911 23472 external_mini_cluster.cc:1428] Setting key a22c88844d239bd07b38c9ad807b094b
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:50.249692 29286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:50.233420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:54679 (local address 127.22.236.62:36777)
0925 15:04:50.233585 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:50.233589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:50.234727 (+ 1138us) server_negotiation.cc:407] Connection header received
0925 15:04:50.236459 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:50.236464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:50.236528 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:50.236610 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:50.237962 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.238656 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.240193 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.240640 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.240730 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:50.244405 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:50.244410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:50.244413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:50.244443 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:50.246762 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:50.247524 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:50.247528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:50.247530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:50.247642 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:50.248076 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:50.248080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:50.248082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:50.248373 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:50.248505 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:50.249151 (+ 646us) server_negotiation.cc:299] Negotiation successful
0925 15:04:50.249363 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:04:50.250681 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:50.233899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36777 (local address 127.22.236.2:54679)
0925 15:04:50.234570 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:50.234611 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:50.236223 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:50.236792 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:50.236799 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:50.237296 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:50.237825 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:50.237852 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.238777 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.238781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:50.240050 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:50.240064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.240853 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.240860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:50.241134 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:50.241850 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:50.241882 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:50.244208 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:50.246884 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:50.246892 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:50.246907 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:50.247428 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:50.247727 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:50.247732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:50.247735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:50.247958 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:50.248491 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:50.248498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:50.248816 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 15:04:50.249049 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 15:04:50.249557 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":269,"threads_started":1}
I20260925 15:04:50.252067 29418 heartbeater.cc:347] Connected to a master server at 127.22.236.62:36777
I20260925 15:04:50.252372 29418 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:50.253039 29418 heartbeater.cc:511] Master 127.22.236.62:36777 requested a full tablet report, sending...
I20260925 15:04:50.254444 29095 ts_manager.cc:194] Registered new tserver with Master: 7f5615eab4b34418a83bac46604c8c5e (127.22.236.2:37717)
I20260925 15:04:50.255594 29095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:54679
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:04:50.281363 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36777
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:50.581791 29423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:50.582063 29423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:50.582180 29423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:50.589265 29423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:50.589371 29423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:50.589519 29423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:04:50.599031 29423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:50.600262 29423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:50.601526 29423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:50.613582 29428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:50.613708 29431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:50.614076 29429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:50.614333 29423 server_base.cc:1034] running on GCE node
I20260925 15:04:50.614934 29423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:50.615824 29423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:50.617082 29423 hybrid_clock.cc:648] HybridClock initialized: now 1790348690617026 us; error 43 us; skew 500 ppm
I20260925 15:04:50.617385 29423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:50.620551 29423 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:04:50.621922 29423 webserver.cc:460] Webserver started at http://127.22.236.3:33389/ using document root <none> and password file <none>
I20260925 15:04:50.622593 29423 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:50.622707 29423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:50.623011 29423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:50.625926 29423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "446d699633b74bc7b12ec4b3a7352d8d"
format_stamp: "Formatted at 2026-09-25 15:04:50 on dist-test-slave-nsrc"
server_key: "04bfddc9d11235516a939c647f077bad"
server_key_iv: "d1633d9a21369aedab6a5b64e85cacb2"
server_key_version: "encryptionkey@0"
I20260925 15:04:50.626685 29423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "446d699633b74bc7b12ec4b3a7352d8d"
format_stamp: "Formatted at 2026-09-25 15:04:50 on dist-test-slave-nsrc"
server_key: "04bfddc9d11235516a939c647f077bad"
server_key_iv: "d1633d9a21369aedab6a5b64e85cacb2"
server_key_version: "encryptionkey@0"
I20260925 15:04:50.631618 29423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 15:04:50.634929 29438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:50.636278 29423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:50.636423 29423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "446d699633b74bc7b12ec4b3a7352d8d"
format_stamp: "Formatted at 2026-09-25 15:04:50 on dist-test-slave-nsrc"
server_key: "04bfddc9d11235516a939c647f077bad"
server_key_iv: "d1633d9a21369aedab6a5b64e85cacb2"
server_key_version: "encryptionkey@0"
I20260925 15:04:50.636611 29423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:50.665517 29423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:50.669502 29423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:50.669725 29423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:50.670570 29423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:50.671767 29423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:50.671836 29423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:50.671931 29423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:50.671980 29423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:50.700569 29423 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:46569
I20260925 15:04:50.700618 29550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:46569 every 8 connection(s)
I20260925 15:04:50.701896 29423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:04:50.707959 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 29423
I20260925 15:04:50.708096 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:04:50.708549 23472 external_mini_cluster.cc:1428] Setting key 2e95f7e3fb381f7b40b9b64e552d5187
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:04:50.719772 29286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:50.704198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:33235 (local address 127.22.236.62:36777)
0925 15:04:50.704366 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:50.704372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:50.705600 (+ 1228us) server_negotiation.cc:407] Connection header received
0925 15:04:50.707326 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:50.707332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:50.707408 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:50.707492 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:50.709020 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.709831 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.711281 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.711657 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.711756 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:50.714780 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:50.714785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:50.714788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:50.714818 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:50.717075 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:50.717728 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:50.717732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:50.717734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:50.717839 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:50.718239 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:50.718243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:50.718245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:50.718497 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:50.718633 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:50.719246 (+ 613us) server_negotiation.cc:299] Negotiation successful
0925 15:04:50.719445 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:04:50.720729 29553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:50.704776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36777 (local address 127.22.236.3:33235)
0925 15:04:50.705450 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:50.705489 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:50.707077 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:50.707729 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:50.707737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:50.708289 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:50.708861 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:50.708878 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.709959 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.709963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:50.711159 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:50.711169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.711845 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.711851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:50.712067 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:50.712645 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:50.712666 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:50.714603 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:50.717216 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:50.717225 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:50.717240 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:50.717620 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:50.717930 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:50.717935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:50.717938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:50.718136 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:50.718608 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:50.718616 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:50.718919 (+ 303us) client_negotiation.cc:769] Sending connection context
0925 15:04:50.719152 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 15:04:50.719652 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":272,"threads_started":1}
I20260925 15:04:50.722184 29551 heartbeater.cc:347] Connected to a master server at 127.22.236.62:36777
I20260925 15:04:50.722481 29551 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:50.723233 29551 heartbeater.cc:511] Master 127.22.236.62:36777 requested a full tablet report, sending...
I20260925 15:04:50.724486 29095 ts_manager.cc:194] Registered new tserver with Master: 446d699633b74bc7b12ec4b3a7352d8d (127.22.236.3:46569)
I20260925 15:04:50.725263 29095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:33235
I20260925 15:04:50.735113 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/test-admin.keytab.
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:50.758977 23472 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 15:04:50.763805 29558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:50.761614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.22.236.1:45371)
0925 15:04:50.762139 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:50.762148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:50.762182 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:04:50.762303 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:50.762310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:50.762515 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:50.762663 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:50.763525 (+ 862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33866: BlockingRecv error: recv got EOF from 127.0.0.1:33866 (error 108)
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":265,"threads_started":1}
W20260925 15:04:50.764271 23472 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/krb5cc)
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:50.766000 23472 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:50.767287 23472 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:04:50.782330 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:50.768994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33870 (local address 127.22.236.1:45371)
0925 15:04:50.769162 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:50.769169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:50.769378 (+ 209us) server_negotiation.cc:407] Connection header received
0925 15:04:50.769557 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:50.769561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:50.769618 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:50.769694 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:50.770626 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.771732 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.773018 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.773539 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.773649 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:50.777216 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:50.777227 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:50.777243 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:50.777281 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:50.779759 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:50.780430 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:50.780437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:50.780439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:50.780557 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:50.780891 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:50.780895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:50.780897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:50.781236 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:50.781492 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:50.781827 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 15:04:50.782080 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 15:04:50.787545 29558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:50.786012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33874 (local address 127.22.236.1:45371)
0925 15:04:50.786202 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:50.786207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:50.786427 (+ 220us) server_negotiation.cc:407] Connection header received
0925 15:04:50.786579 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:50.786583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:50.786634 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:50.786709 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:50.787380 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33874: BlockingRecv error: recv got EOF from 127.0.0.1:33874 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 15:04:50.787793 23472 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/krb5cc' of type FILE
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:50.789317 23472 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:50.790524 23472 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:04:50.799010 29284 heartbeater.cc:503] Master 127.22.236.62:36777 was elected leader, sending a full tablet report...
I20260925 15:04:50.803316 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:50.791668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33888 (local address 127.22.236.1:45371)
0925 15:04:50.791834 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:50.791838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:50.792137 (+ 299us) server_negotiation.cc:407] Connection header received
0925 15:04:50.792321 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:50.792325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:50.792389 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:50.792466 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:50.793385 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.794046 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.795124 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.795538 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.795635 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:50.798724 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:50.798728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:50.798731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:50.798759 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:50.801370 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:50.801889 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:50.801893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:50.801895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:50.802012 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:50.802289 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:50.802291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:50.802293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:50.802541 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:50.802715 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:50.802896 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 15:04:50.803037 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 15:04:50.807029 29558 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:50.805587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33892 (local address 127.22.236.1:45371)
0925 15:04:50.805751 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:50.805756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:50.806007 (+ 251us) server_negotiation.cc:407] Connection header received
0925 15:04:50.806081 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:50.806084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:50.806144 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:50.806212 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:50.806850 (+ 638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33892: BlockingRecv error: recv got EOF from 127.0.0.1:33892 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 15:04:50.807164 23472 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348645297121-23472-0/krb5kdc/krb5cc' of type FILE
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:50.808601 23472 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:50.809769 23472 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:04:50.823292 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:50.811032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.22.236.1:45371)
0925 15:04:50.811215 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:50.811221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:50.811478 (+ 257us) server_negotiation.cc:407] Connection header received
0925 15:04:50.811616 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:50.811621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:50.811677 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:50.811751 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:50.812745 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.813470 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.814707 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.815129 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.815234 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:50.818677 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:50.818682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:50.818685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:50.818716 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:50.820941 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:50.821515 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:50.821519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:50.821522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:50.821648 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:50.821995 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:50.822000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:50.822002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:50.822261 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:50.822554 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:50.822794 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 15:04:50.822996 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:04:50.824882 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 29156
I20260925 15:04:50.836304 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:50.825537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33916 (local address 127.22.236.1:45371)
0925 15:04:50.825679 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:50.825684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:50.826011 (+ 327us) server_negotiation.cc:407] Connection header received
0925 15:04:50.826145 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:50.826151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:50.826209 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:50.826281 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:50.827126 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.827792 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.829044 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.829464 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.829562 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:50.831446 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:50.831451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:50.831454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:50.831481 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:50.833686 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:50.834239 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:50.834244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:50.834248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:50.834366 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:50.834690 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:50.834695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:50.834699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:50.835061 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:50.835341 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:50.835633 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:04:50.835872 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:04:50.836874 29279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:50.917425 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 29156
I20260925 15:04:50.929668 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 29290
Sep 25 15:04:50 dist-test-slave-nsrc krb5kdc[29050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:04:50.944264 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:50.930500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.22.236.2:37717)
0925 15:04:50.930968 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:50.930975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:50.930999 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:04:50.931057 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:50.931063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:50.931268 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:50.931395 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:50.932377 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.933356 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.934532 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:50.935076 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:50.935172 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:50.938946 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:50.938957 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:50.938971 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:50.939001 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:50.941656 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:50.942304 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:50.942311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:50.942313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:50.942433 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:50.942757 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:50.942761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:50.942763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:50.943081 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:50.943357 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:50.943695 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 15:04:50.943940 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20260925 15:04:50.945416 29413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:51.024379 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 29290
I20260925 15:04:51.036397 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 29423
Sep 25 15:04:51 dist-test-slave-nsrc krb5kdc[29050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:04:51.050484 29573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:51.037363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.22.236.3:46569)
0925 15:04:51.037945 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:51.037952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:51.037978 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:04:51.038043 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:51.038049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:51.038269 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:51.038387 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:51.039298 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:51.040359 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:51.041595 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:51.042113 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:51.042197 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:51.045796 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:51.045806 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:51.045819 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:51.045849 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:51.048414 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:51.048991 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:51.049005 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:51.049008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:51.049113 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:51.049383 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:51.049385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:51.049387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:51.049664 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:51.049879 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:51.050055 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 15:04:51.050225 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":308,"threads_started":1}
I20260925 15:04:51.051584 29546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:51.133643 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 29423
I20260925 15:04:51.145180 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 29063
I20260925 15:04:51.146211 29125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:51.245043 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 29063
2026-09-25T15:04:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3642 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:04:51 dist-test-slave-nsrc krb5kdc[29578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:04:51 dist-test-slave-nsrc krb5kdc[29578](info): set up 2 sockets
Sep 25 15:04:51 dist-test-slave-nsrc krb5kdc[29578](info): commencing operation
krb5kdc: starting...
W20260925 15:04:52.244565 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:04:52 dist-test-slave-nsrc krb5kdc[29578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:04:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:04:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.62.keytab.
Entry for principal oryx/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.62.keytab.
I20260925 15:04:52.425549 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:45627
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.62.keytab
--principal=oryx/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:45627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:52.731110 29591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:52.731406 29591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:52.731534 29591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:52.739151 29591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:52.739264 29591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:52.739338 29591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:52.739418 29591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:52.739467 29591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:04:52.749255 29591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:45627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.62.keytab
--principal=oryx/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:45627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:52.750514 29591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:52.751781 29591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:52.762893 29596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:52.762948 29599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:52.762919 29597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:52.763528 29591 server_base.cc:1034] running on GCE node
I20260925 15:04:52.764460 29591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:52.765920 29591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:52.767122 29591 hybrid_clock.cc:648] HybridClock initialized: now 1790348692767066 us; error 48 us; skew 500 ppm
I20260925 15:04:52.767436 29591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:52 dist-test-slave-nsrc krb5kdc[29578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348692, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:52.770247 29591 init.cc:377] Logged in from keytab as oryx/127.22.236.62@KRBTEST.COM (short username oryx)
I20260925 15:04:52.771595 29591 webserver.cc:460] Webserver started at http://127.22.236.62:45137/ using document root <none> and password file <none>
I20260925 15:04:52.772250 29591 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:52.772373 29591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:52.772639 29591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:52.775408 29591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "d5cdd435e9804a6c820218e27b67d29d"
format_stamp: "Formatted at 2026-09-25 15:04:52 on dist-test-slave-nsrc"
server_key: "6927c24ec69c5662e49a18a60eb37d05"
server_key_iv: "54496b07a280bad453416195dc799229"
server_key_version: "encryptionkey@0"
I20260925 15:04:52.776091 29591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "d5cdd435e9804a6c820218e27b67d29d"
format_stamp: "Formatted at 2026-09-25 15:04:52 on dist-test-slave-nsrc"
server_key: "6927c24ec69c5662e49a18a60eb37d05"
server_key_iv: "54496b07a280bad453416195dc799229"
server_key_version: "encryptionkey@0"
I20260925 15:04:52.780881 29591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:04:52.784018 29606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:52.785297 29591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:04:52.785451 29591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "d5cdd435e9804a6c820218e27b67d29d"
format_stamp: "Formatted at 2026-09-25 15:04:52 on dist-test-slave-nsrc"
server_key: "6927c24ec69c5662e49a18a60eb37d05"
server_key_iv: "54496b07a280bad453416195dc799229"
server_key_version: "encryptionkey@0"
I20260925 15:04:52.785629 29591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:52.822614 29591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:52.826642 29591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:52.826907 29591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:52.843102 29591 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:45627
I20260925 15:04:52.843144 29657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:45627 every 8 connection(s)
I20260925 15:04:52.844377 29591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:04:52.847821 29658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:52.853062 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 29591
I20260925 15:04:52.853255 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:04:52.854035 23472 external_mini_cluster.cc:1428] Setting key 430de864ecb67c48ceb0328c2499572f
I20260925 15:04:52.858599 29658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d: Bootstrap starting.
I20260925 15:04:52.861433 29658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:52.862385 29658 log.cc:824] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:52.865401 29658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d: No bootstrap required, opened a new log
Sep 25 15:04:52 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
I20260925 15:04:52.874820 29658 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } }
I20260925 15:04:52.875267 29658 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:52.875398 29658 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5cdd435e9804a6c820218e27b67d29d, State: Initialized, Role: FOLLOWER
I20260925 15:04:52.876052 29658 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } }
I20260925 15:04:52.876277 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:52.856068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45218 (local address 127.22.236.62:45627)
0925 15:04:52.856798 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:52.856810 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:52.856849 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:04:52.858413 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:52.858434 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:52.858826 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:52.859170 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:52.860559 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:52.861839 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:52.863198 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:52.864180 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:52.864371 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:52.868082 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:52.868092 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:52.868111 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:52.868198 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:52.871977 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:52.872643 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:52.872651 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:52.872660 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:52.872873 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:52.873187 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:52.873195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:52.873199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:52.873926 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:52.874207 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:52.874491 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 15:04:52.874908 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":537,"thread_start_us":317,"threads_started":1}
I20260925 15:04:52.876341 29658 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:04:52.876477 29658 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:04:52.876638 29658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:52.878821 29658 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } }
I20260925 15:04:52.879263 29658 leader_election.cc:302] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5cdd435e9804a6c820218e27b67d29d; no voters:
I20260925 15:04:52.879779 29658 leader_election.cc:288] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:04:52.879884 29663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:52.880508 29663 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [term 1 LEADER]: Becoming Leader. State: Replica: d5cdd435e9804a6c820218e27b67d29d, State: Running, Role: LEADER
I20260925 15:04:52.880963 29663 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } }
I20260925 15:04:52.881623 29658 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:52.883602 29664 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } }
I20260925 15:04:52.883953 29664 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:52.884384 29665 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5cdd435e9804a6c820218e27b67d29d. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } }
I20260925 15:04:52.884759 29665 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:52.889523 29668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:52.893611 29668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:52.902001 29668 catalog_manager.cc:1347] Generated new cluster ID: d8f4c53ce2424dabaec159f4fffbc069
I20260925 15:04:52.902117 29668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:52.921663 29668 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:52.923002 29668 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:52.955240 29668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d: Generated new TSK 0
I20260925 15:04:52.956171 29668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.1.keytab.
Entry for principal oryx/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.1.keytab.
I20260925 15:04:53.019392 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:45627
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.1.keytab
--principal=oryx/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:53.328014 29684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:53.328332 29684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:53.328472 29684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:53.335578 29684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:53.335693 29684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:53.335852 29684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:04:53.345495 29684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.1.keytab
--principal=oryx/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:45627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:53.346725 29684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:53.348017 29684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:53.360359 29692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:53.360394 29690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:53.360414 29689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:53.361054 29684 server_base.cc:1034] running on GCE node
I20260925 15:04:53.361598 29684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:53.362465 29684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:53.363682 29684 hybrid_clock.cc:648] HybridClock initialized: now 1790348693363614 us; error 47 us; skew 500 ppm
I20260925 15:04:53.363966 29684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:53 dist-test-slave-nsrc krb5kdc[29578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348693, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:53.366887 29684 init.cc:377] Logged in from keytab as oryx/127.22.236.1@KRBTEST.COM (short username oryx)
I20260925 15:04:53.368208 29684 webserver.cc:460] Webserver started at http://127.22.236.1:39043/ using document root <none> and password file <none>
I20260925 15:04:53.368835 29684 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:53.368942 29684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:53.369203 29684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:53.371912 29684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "3ba42b62df1045df88a14643e5ac471a"
format_stamp: "Formatted at 2026-09-25 15:04:53 on dist-test-slave-nsrc"
server_key: "0341972dcd22775ed09f123281d3e031"
server_key_iv: "06e549948ac898e77079795d59996896"
server_key_version: "encryptionkey@0"
I20260925 15:04:53.372670 29684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "3ba42b62df1045df88a14643e5ac471a"
format_stamp: "Formatted at 2026-09-25 15:04:53 on dist-test-slave-nsrc"
server_key: "0341972dcd22775ed09f123281d3e031"
server_key_iv: "06e549948ac898e77079795d59996896"
server_key_version: "encryptionkey@0"
I20260925 15:04:53.377635 29684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:04:53.380769 29699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:53.382084 29684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:53.382222 29684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "3ba42b62df1045df88a14643e5ac471a"
format_stamp: "Formatted at 2026-09-25 15:04:53 on dist-test-slave-nsrc"
server_key: "0341972dcd22775ed09f123281d3e031"
server_key_iv: "06e549948ac898e77079795d59996896"
server_key_version: "encryptionkey@0"
I20260925 15:04:53.382381 29684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:53.398654 29684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:53.402585 29684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:53.402833 29684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:53.403729 29684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:53.405035 29684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:53.405109 29684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:53.405221 29684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:53.405275 29684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:53.433498 29684 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:46247
I20260925 15:04:53.433537 29811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:46247 every 8 connection(s)
I20260925 15:04:53.434670 29684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:04:53.436616 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 29684
I20260925 15:04:53.436786 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:04:53.437536 23472 external_mini_cluster.cc:1428] Setting key 296bbd07e7085d74fab53818abf9ca1b
Sep 25 15:04:53 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348693, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
I20260925 15:04:53.453091 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:53.437177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:47945 (local address 127.22.236.62:45627)
0925 15:04:53.437618 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:53.437624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:53.438535 (+ 911us) server_negotiation.cc:407] Connection header received
0925 15:04:53.440252 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:53.440259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:53.440342 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:53.440448 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:53.441776 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:53.442721 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:53.444177 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:53.444641 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:53.444732 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:53.448188 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:53.448195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:53.448198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:53.448227 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:53.450399 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:53.451026 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:53.451032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:53.451035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:53.451142 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:53.451508 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:53.451513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:53.451517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:53.451774 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:53.451894 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:53.452544 (+ 650us) server_negotiation.cc:299] Negotiation successful
0925 15:04:53.452754 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":214,"threads_started":1}
I20260925 15:04:53.453987 29815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:53.437637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.1:47945)
0925 15:04:53.438382 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:53.438431 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:53.440001 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:53.440629 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:53.440636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:53.441117 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:53.441633 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:53.441648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:53.442841 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:53.442844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:53.444036 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:53.444047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:53.444855 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:53.444862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:53.445142 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:53.445824 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:53.445846 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:53.447965 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:53.450527 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:53.450536 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:53.450551 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:53.450916 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:53.451234 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:53.451239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:53.451243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:53.451409 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:53.451884 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:53.451891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:53.452226 (+ 335us) client_negotiation.cc:769] Sending connection context
0925 15:04:53.452454 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 15:04:53.452922 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":302,"threads_started":1}
WARNING: no policy specified for oryx/127.22.236.2@KRBTEST.COM; defaulting to no policy
I20260925 15:04:53.455518 29812 heartbeater.cc:347] Connected to a master server at 127.22.236.62:45627
I20260925 15:04:53.455794 29812 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:53.456568 29812 heartbeater.cc:511] Master 127.22.236.62:45627 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.236.2@KRBTEST.COM" created.
I20260925 15:04:53.458729 29623 ts_manager.cc:194] Registered new tserver with Master: 3ba42b62df1045df88a14643e5ac471a (127.22.236.1:46247)
I20260925 15:04:53.460493 29623 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:47945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.2.keytab.
Entry for principal oryx/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.2.keytab.
I20260925 15:04:53.479094 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:45627
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.2.keytab
--principal=oryx/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:53.783064 29818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:53.783324 29818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:53.783438 29818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:53.790563 29818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:53.790707 29818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:53.790915 29818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:04:53.800550 29818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.2.keytab
--principal=oryx/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:45627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:53.801894 29818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:53.803238 29818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:53.815328 29823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:53.815279 29824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:53.815352 29826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:53.816080 29818 server_base.cc:1034] running on GCE node
I20260925 15:04:53.816614 29818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:53.817441 29818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:53.818626 29818 hybrid_clock.cc:648] HybridClock initialized: now 1790348693818578 us; error 44 us; skew 500 ppm
I20260925 15:04:53.818897 29818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:53 dist-test-slave-nsrc krb5kdc[29578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348693, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:53.821763 29818 init.cc:377] Logged in from keytab as oryx/127.22.236.2@KRBTEST.COM (short username oryx)
I20260925 15:04:53.823047 29818 webserver.cc:460] Webserver started at http://127.22.236.2:36319/ using document root <none> and password file <none>
I20260925 15:04:53.823625 29818 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:53.823717 29818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:53.823951 29818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:53.826642 29818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "0b635f83f59d4b2b9f169b982f49c3f5"
format_stamp: "Formatted at 2026-09-25 15:04:53 on dist-test-slave-nsrc"
server_key: "70f03adf0692a58b21fc386d24fb9780"
server_key_iv: "d619e49fff9331903adb1934fe7d7ba5"
server_key_version: "encryptionkey@0"
I20260925 15:04:53.827317 29818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "0b635f83f59d4b2b9f169b982f49c3f5"
format_stamp: "Formatted at 2026-09-25 15:04:53 on dist-test-slave-nsrc"
server_key: "70f03adf0692a58b21fc386d24fb9780"
server_key_iv: "d619e49fff9331903adb1934fe7d7ba5"
server_key_version: "encryptionkey@0"
I20260925 15:04:53.831943 29818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260925 15:04:53.835028 29833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:53.836325 29818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:53.836449 29818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "0b635f83f59d4b2b9f169b982f49c3f5"
format_stamp: "Formatted at 2026-09-25 15:04:53 on dist-test-slave-nsrc"
server_key: "70f03adf0692a58b21fc386d24fb9780"
server_key_iv: "d619e49fff9331903adb1934fe7d7ba5"
server_key_version: "encryptionkey@0"
I20260925 15:04:53.836601 29818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:53.863370 29818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:53.867287 29818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:53.867516 29818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:53.868376 29818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:53.869557 29818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:53.869624 29818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:53.869710 29818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:53.869751 29818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:53.898536 29818 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:38433
I20260925 15:04:53.898607 29945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:38433 every 8 connection(s)
I20260925 15:04:53.899737 29818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:04:53.905617 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 29818
I20260925 15:04:53.905771 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:04:53.906152 23472 external_mini_cluster.cc:1428] Setting key 5ada10f52cb88fa10bd612470ed1bdaa
Sep 25 15:04:53 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348693, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
I20260925 15:04:53.917858 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:53.901937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:47583 (local address 127.22.236.62:45627)
0925 15:04:53.902133 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:53.902138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:53.903205 (+ 1067us) server_negotiation.cc:407] Connection header received
0925 15:04:53.904957 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:53.904961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:53.905041 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:53.905116 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:53.906605 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:53.907493 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:53.909066 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:53.909648 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:53.909765 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:53.912810 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:53.912817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:53.912821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:53.912859 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:53.915171 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:53.915937 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:53.915941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:53.915943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:53.916054 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:53.916458 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:53.916461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:53.916463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:53.916692 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:53.916816 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:53.917481 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 15:04:53.917638 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:04:53.919159 29948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:53.902414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.2:47583)
0925 15:04:53.903058 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:53.903095 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:53.904701 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:53.905292 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:53.905300 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:53.905869 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:53.906445 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:53.906463 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:53.907622 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:53.907626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:53.908930 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:53.908942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:53.909848 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:53.909853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:53.910038 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:53.910628 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:53.910648 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:53.912604 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:53.915294 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:53.915302 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:53.915319 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:53.915839 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:53.916137 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:53.916160 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:53.916164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:53.916367 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:53.916802 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:53.916810 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:53.917162 (+ 352us) client_negotiation.cc:769] Sending connection context
0925 15:04:53.917417 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 15:04:53.917892 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":244,"threads_started":1}
I20260925 15:04:53.920418 29946 heartbeater.cc:347] Connected to a master server at 127.22.236.62:45627
I20260925 15:04:53.920707 29946 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:53.921353 29946 heartbeater.cc:511] Master 127.22.236.62:45627 requested a full tablet report, sending...
I20260925 15:04:53.922623 29623 ts_manager.cc:194] Registered new tserver with Master: 0b635f83f59d4b2b9f169b982f49c3f5 (127.22.236.2:38433)
I20260925 15:04:53.923362 29623 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:47583
WARNING: no policy specified for oryx/127.22.236.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.236.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.3.keytab.
Entry for principal oryx/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.3.keytab.
I20260925 15:04:53.946745 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:45627
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.3.keytab
--principal=oryx/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:54.245427 29951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:54.245697 29951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:54.245826 29951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:54.252794 29951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:54.252911 29951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:54.253067 29951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:04:54.262315 29951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.3.keytab
--principal=oryx/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:45627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:54.263599 29951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:54.264930 29951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:54.277001 29957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:54.276966 29959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:54.277032 29956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:54.277796 29951 server_base.cc:1034] running on GCE node
I20260925 15:04:54.278373 29951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:54.279201 29951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:54.280388 29951 hybrid_clock.cc:648] HybridClock initialized: now 1790348694280341 us; error 40 us; skew 500 ppm
I20260925 15:04:54.280669 29951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:54 dist-test-slave-nsrc krb5kdc[29578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348694, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:54.283681 29951 init.cc:377] Logged in from keytab as oryx/127.22.236.3@KRBTEST.COM (short username oryx)
I20260925 15:04:54.285017 29951 webserver.cc:460] Webserver started at http://127.22.236.3:39719/ using document root <none> and password file <none>
I20260925 15:04:54.285684 29951 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:54.285792 29951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:54.286089 29951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:54.288911 29951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "cd6d4fd224df40d48f64c3e8ee9446d7"
format_stamp: "Formatted at 2026-09-25 15:04:54 on dist-test-slave-nsrc"
server_key: "1480d366c37ef7145a1304b9260e0caa"
server_key_iv: "245ee94364c01d0d0c8f010ebe00c851"
server_key_version: "encryptionkey@0"
I20260925 15:04:54.289675 29951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "cd6d4fd224df40d48f64c3e8ee9446d7"
format_stamp: "Formatted at 2026-09-25 15:04:54 on dist-test-slave-nsrc"
server_key: "1480d366c37ef7145a1304b9260e0caa"
server_key_iv: "245ee94364c01d0d0c8f010ebe00c851"
server_key_version: "encryptionkey@0"
I20260925 15:04:54.294520 29951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:04:54.297935 29966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:54.299196 29951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 15:04:54.299332 29951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "cd6d4fd224df40d48f64c3e8ee9446d7"
format_stamp: "Formatted at 2026-09-25 15:04:54 on dist-test-slave-nsrc"
server_key: "1480d366c37ef7145a1304b9260e0caa"
server_key_iv: "245ee94364c01d0d0c8f010ebe00c851"
server_key_version: "encryptionkey@0"
I20260925 15:04:54.299504 29951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:54.318183 29951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:54.322252 29951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:54.322507 29951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:54.323364 29951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:54.324621 29951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:54.324694 29951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:54.324820 29951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:54.324880 29951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:54.352596 29951 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:35359
I20260925 15:04:54.352650 30078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:35359 every 8 connection(s)
I20260925 15:04:54.353778 29951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:04:54.362182 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 29951
I20260925 15:04:54.362344 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:04:54.362742 23472 external_mini_cluster.cc:1428] Setting key 3eaaf94ce954dd3e70392e930c242680
Sep 25 15:04:54 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348694, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
I20260925 15:04:54.371901 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:54.356109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:34933 (local address 127.22.236.62:45627)
0925 15:04:54.356331 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:54.356337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:54.357330 (+ 993us) server_negotiation.cc:407] Connection header received
0925 15:04:54.359020 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:54.359024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:54.359088 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:54.359170 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:54.360691 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.361435 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.363070 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.363566 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.363681 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:54.367034 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:54.367039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:54.367042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:54.367072 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:54.369428 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:54.370038 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:54.370041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:54.370043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:54.370144 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:54.370551 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:54.370554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:54.370556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:54.370784 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:54.370888 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:54.371502 (+ 614us) server_negotiation.cc:299] Negotiation successful
0925 15:04:54.371662 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 15:04:54.372956 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:54.356537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.3:34933)
0925 15:04:54.357187 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:54.357222 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:54.358761 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:54.359371 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:54.359380 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:54.359852 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:54.360550 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:54.360571 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.361552 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.361555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:54.362911 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:54.362923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.363778 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.363785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:54.364073 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:54.364720 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:54.364747 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:54.366856 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:54.369540 (+ 2684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:54.369546 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:54.369558 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:54.369943 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:54.370253 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:54.370257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:54.370260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:54.370455 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:54.370879 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:54.370886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:54.371197 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 15:04:54.371434 (+ 237us) client_negotiation.cc:240] Negotiation successful
0925 15:04:54.371817 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":241,"threads_started":1}
I20260925 15:04:54.374341 30079 heartbeater.cc:347] Connected to a master server at 127.22.236.62:45627
I20260925 15:04:54.374607 30079 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:54.375322 30079 heartbeater.cc:511] Master 127.22.236.62:45627 requested a full tablet report, sending...
I20260925 15:04:54.376782 29622 ts_manager.cc:194] Registered new tserver with Master: cd6d4fd224df40d48f64c3e8ee9446d7 (127.22.236.3:35359)
I20260925 15:04:54.377779 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:04:54.377790 29622 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:34933
WARNING: no policy specified for oryx/127.22.236.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.236.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.236.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.61.keytab.
Entry for principal oryx/127.22.236.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.61.keytab.
I20260925 15:04:54.420758 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.61:40973
--webserver_interface=127.22.236.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.61.keytab
--principal=oryx/127.22.236.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:45627,127.22.236.61:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:04:54.463740 29812 heartbeater.cc:503] Master 127.22.236.62:45627 was elected leader, sending a full tablet report...
W20260925 15:04:54.727396 30084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:54.727685 30084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:54.727818 30084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:54.734974 30084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:54.735100 30084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:54.735178 30084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:54.735252 30084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:54.735317 30084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:04:54.745481 30084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:45627,127.22.236.61:40973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.61.keytab
--principal=oryx/127.22.236.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.61:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.236.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
W20260925 15:04:54.745829 30084 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.22.236.62:45627,127.22.236.61:40973'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 15:04:54.746791 30084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:54.748003 30084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:54.758780 30089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:54.758828 30092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:54.758852 30090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:54.759598 30084 server_base.cc:1034] running on GCE node
I20260925 15:04:54.760352 30084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:54.761658 30084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:54.762907 30084 hybrid_clock.cc:648] HybridClock initialized: now 1790348694762837 us; error 43 us; skew 500 ppm
I20260925 15:04:54.763201 30084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:54 dist-test-slave-nsrc krb5kdc[29578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348694, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:54.771328 30084 init.cc:377] Logged in from keytab as oryx/127.22.236.61@KRBTEST.COM (short username oryx)
I20260925 15:04:54.772719 30084 webserver.cc:460] Webserver started at http://127.22.236.61:37429/ using document root <none> and password file <none>
I20260925 15:04:54.773402 30084 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:54.773491 30084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:54.773772 30084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:54.776648 30084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/data/instance:
uuid: "69619aa463594ba086f18ec78c678cc4"
format_stamp: "Formatted at 2026-09-25 15:04:54 on dist-test-slave-nsrc"
server_key: "0518037be96ccfc32b68e11b4b2f553f"
server_key_iv: "733bc4f99cd95a48b6d9941fcce81060"
server_key_version: "encryptionkey@0"
I20260925 15:04:54.777367 30084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/wal/instance:
uuid: "69619aa463594ba086f18ec78c678cc4"
format_stamp: "Formatted at 2026-09-25 15:04:54 on dist-test-slave-nsrc"
server_key: "0518037be96ccfc32b68e11b4b2f553f"
server_key_iv: "733bc4f99cd95a48b6d9941fcce81060"
server_key_version: "encryptionkey@0"
I20260925 15:04:54.782119 30084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:04:54.785348 30099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:54.786728 30084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:04:54.786909 30084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/wal
uuid: "69619aa463594ba086f18ec78c678cc4"
format_stamp: "Formatted at 2026-09-25 15:04:54 on dist-test-slave-nsrc"
server_key: "0518037be96ccfc32b68e11b4b2f553f"
server_key_iv: "733bc4f99cd95a48b6d9941fcce81060"
server_key_version: "encryptionkey@0"
I20260925 15:04:54.787082 30084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:54.803717 30084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:54.807510 30084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:54.807739 30084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:54.824324 30084 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.61:40973
I20260925 15:04:54.824343 30150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.61:40973 every 8 connection(s)
I20260925 15:04:54.825614 30084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/data/info.pb
I20260925 15:04:54.827008 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 30084
I20260925 15:04:54.829380 30151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:54.836128 30151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:04:54 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.236.61@KRBTEST.COM
Sep 25 15:04:54 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348694, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.61@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
I20260925 15:04:54.848696 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:54.829011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.22.236.61:40973)
0925 15:04:54.829689 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:54.829700 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:54.829739 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:04:54.831247 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:54.831266 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:54.831632 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:54.831994 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:54.833488 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.834802 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.835961 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.836668 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.836836 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:54.841060 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:54.841066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:54.841076 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:54.841107 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:54.844357 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:54.845092 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:54.845099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:54.845107 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:54.845320 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:54.845677 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:54.845682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:54.845688 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:54.846315 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:54.846579 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:54.847016 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 15:04:54.847489 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":302,"threads_started":1}
I20260925 15:04:54.854032 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:54.838592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45230 (local address 127.22.236.62:45627)
0925 15:04:54.838756 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:54.838762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:54.839223 (+ 461us) server_negotiation.cc:407] Connection header received
0925 15:04:54.839440 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:54.839444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:54.839505 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:54.839585 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:54.840752 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.841587 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.843515 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.844179 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.844290 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:54.847846 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:54.847853 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:54.847858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:54.847919 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:54.851974 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:54.852598 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:54.852602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:54.852604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:54.852705 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:54.852963 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:54.852966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:54.852967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:54.853243 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:54.853342 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:54.853685 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 15:04:54.853826 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:04:54.854071 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:54.838515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:45230)
0925 15:04:54.839098 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:54.839129 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:54.839270 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:54.839653 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:54.839659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:54.840357 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:54.840622 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:54.840631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.841709 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.841714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:54.843350 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:54.843368 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.844352 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.844357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:54.844534 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:54.845064 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:54.845086 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:54.847654 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:54.852088 (+ 4434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:54.852095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:54.852099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:54.852505 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:54.852777 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:54.852780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:54.852782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:54.852883 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:54.853351 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:54.853356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:54.853479 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 15:04:54.853619 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 15:04:54.853821 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":267,"threads_started":1}
I20260925 15:04:54.855509 30151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:04:54 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348694, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.61@KRBTEST.COM for oryx/127.22.236.61@KRBTEST.COM
I20260925 15:04:54.866089 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:54.856224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40973 (local address 127.0.0.1:35362)
0925 15:04:54.856387 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:54.856403 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:54.856498 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:54.856747 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:54.856751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:54.857051 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:54.857161 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:54.857168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.858045 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.858050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:54.858846 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:54.858856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.859446 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.859451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:54.859622 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:54.860063 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:54.860077 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:54.861788 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:54.864086 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:54.864091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:54.864095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:54.864437 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:54.864742 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:54.864746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:54.864750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:54.864868 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:54.865281 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:54.865286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:54.865412 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 15:04:54.865509 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:04:54.865751 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 15:04:54.866250 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:54.856231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.22.236.61:40973)
0925 15:04:54.856373 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:54.856379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:54.856459 (+ 80us) server_negotiation.cc:407] Connection header received
0925 15:04:54.856595 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:54.856600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:54.856649 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:54.856716 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:54.857271 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.857920 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.858962 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.859329 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.859422 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:54.861928 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:54.861932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:54.861935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:54.861959 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:54.863979 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:54.864536 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:54.864539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:54.864541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:54.864652 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:54.864962 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:54.864965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:54.864967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:54.865187 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:54.865297 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:54.865613 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 15:04:54.865902 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":37}
I20260925 15:04:54.870795 30151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: Bootstrap starting.
I20260925 15:04:54.873797 30151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:54.874748 30151 log.cc:824] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:54.877171 30151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: No bootstrap required, opened a new log
I20260925 15:04:54.883790 30151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } }
I20260925 15:04:54.884081 30151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:54.884179 30151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69619aa463594ba086f18ec78c678cc4, State: Initialized, Role: FOLLOWER
I20260925 15:04:54.884696 30151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } }
I20260925 15:04:54.885473 30157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } } }
I20260925 15:04:54.885761 30157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:04:54.886528 30151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:04:54.894151 30168 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:04:54.894258 30168 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:04:54.896517 30084 master_runner.cc:428] Detected that this master 69619aa463594ba086f18ec78c678cc4 is joining an existing cluster
I20260925 15:04:54.896592 30084 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.236.61:40973
I20260925 15:04:54.925789 29946 heartbeater.cc:503] Master 127.22.236.62:45627 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.22.236.60@KRBTEST.COM; defaulting to no policy
I20260925 15:04:54.928234 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:54.902944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.22.236.62:45627)
0925 15:04:54.903132 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:54.903138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:54.912792 (+ 9654us) server_negotiation.cc:407] Connection header received
0925 15:04:54.913054 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:54.913060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:54.913145 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:54.913214 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:54.914637 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.915717 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.917543 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.918103 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.918191 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:54.920644 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:54.920651 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:54.920656 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:54.920694 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:54.923636 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:54.925959 (+ 2323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:54.925965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:54.925968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:54.926116 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:54.926556 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:54.926561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:54.926565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:54.926903 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:54.927031 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:54.927636 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 15:04:54.927867 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 15:04:54.928965 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:54.903038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:45242)
0925 15:04:54.912676 (+ 9638us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:54.912698 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:54.912911 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:54.913434 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:54.913440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:54.914126 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:54.914473 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:54.914485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.915879 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.915883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:54.917382 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:54.917394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.918442 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.918447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:54.918666 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:54.919291 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:54.919314 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:54.920441 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:54.925319 (+ 4878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:54.925325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:54.925328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:54.925810 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:54.926256 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:54.926261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:54.926265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:54.926431 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:54.927106 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:54.927111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:54.927306 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 15:04:54.928128 (+ 822us) client_negotiation.cc:240] Negotiation successful
0925 15:04:54.928482 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9452,"mutex_wait_us":35,"thread_start_us":10899,"threads_started":1}
I20260925 15:04:54.929829 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:54.909022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40973 (local address 127.0.0.1:35366)
0925 15:04:54.914273 (+ 5251us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:54.914290 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:54.914438 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:54.914868 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:54.914872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:54.915514 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:54.915813 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:54.915825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.917268 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.917273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:54.918840 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:54.918858 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.919938 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.919942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:54.920071 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:54.920647 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:54.920670 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:54.921606 (+ 936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:54.925328 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:54.925333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:54.925337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:54.925957 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:54.927915 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:54.927921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:54.927924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:54.928076 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:54.928768 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:54.928773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:54.928902 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 15:04:54.929039 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 15:04:54.929313 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5102,"mutex_wait_us":3,"spinlock_wait_cycles":27392,"thread_start_us":3298,"threads_started":1}
I20260925 15:04:54.930085 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:54.908542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.22.236.61:40973)
0925 15:04:54.908718 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:54.908724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:54.914391 (+ 5667us) server_negotiation.cc:407] Connection header received
0925 15:04:54.914568 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:54.914573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:54.914637 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:54.914697 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:54.915958 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.917074 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.918988 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.919624 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.919732 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:54.921757 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:54.921762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:54.921767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:54.921793 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:54.925181 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:54.926067 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:54.926072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:54.926074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:54.926220 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:54.928201 (+ 1981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:54.928205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:54.928208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:54.928555 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:54.928685 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:54.929175 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 15:04:54.929467 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.236.60@KRBTEST.COM" created.
Sep 25 15:04:54 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348692, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.62@KRBTEST.COM for oryx/127.22.236.61@KRBTEST.COM
I20260925 15:04:54.949826 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:54.934792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.22.236.61:40973)
0925 15:04:54.934947 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:54.934952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:54.935361 (+ 409us) server_negotiation.cc:407] Connection header received
0925 15:04:54.935657 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:54.935662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:54.935745 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:54.935819 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:54.936905 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.937609 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.938836 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.939472 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.939605 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:54.943431 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:54.943437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:54.943441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:54.943472 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:54.946827 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:54.947632 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:54.947637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:54.947641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:54.947797 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:54.948211 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:54.948216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:54.948219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:54.948601 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:54.948737 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:54.949246 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 15:04:54.949462 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:04:54.949857 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:54.934695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40973 (local address 127.0.0.1:35382)
0925 15:04:54.935251 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:54.935285 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:54.935475 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:54.935877 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:54.935884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:54.936538 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:54.936768 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:54.936777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.937717 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.937723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:54.938700 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:54.938711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:54.939650 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:54.939657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:54.939875 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:54.940480 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:54.940504 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:54.943224 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:54.946950 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:54.946962 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:54.946970 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:54.947511 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:54.947915 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:54.947920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:54.947923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:54.948073 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:54.948729 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:54.948737 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:54.948933 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 15:04:54.949126 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 15:04:54.949448 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":204,"threads_started":1}
I20260925 15:04:54.951954 29622 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.236.61:40973: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "69619aa463594ba086f18ec78c678cc4"
member_type: NON_VOTER
last_known_addr {
host: "127.22.236.61"
port: 40973
}
attrs {
promote: true
}
}
dest_uuid: "d5cdd435e9804a6c820218e27b67d29d"
cas_config_opid_index: -1
I20260925 15:04:54.953061 29622 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } }
I20260925 15:04:54.956813 30126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:54.956436 30180 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 69619aa463594ba086f18ec78c678cc4 (127.22.236.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.236.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.60.keytab.
Entry for principal oryx/127.22.236.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.60.keytab.
I20260925 15:04:54.959288 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.60:38781
--webserver_interface=127.22.236.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.60.keytab
--principal=oryx/127.22.236.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:45627,127.22.236.61:40973,127.22.236.60:38781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:04:54.960427 30126 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 FOLLOWER]: Refusing update from remote peer d5cdd435e9804a6c820218e27b67d29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 15:04:54.969668 30181 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:54.971302 30180 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.236.61:40973
I20260925 15:04:54.971832 30182 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } } }
I20260925 15:04:54.972133 30182 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:54.976938 30084 master.cc:537] Master@127.22.236.61:40973 shutting down...
I20260925 15:04:55.000645 30126 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 69619aa463594ba086f18ec78c678cc4 (127.22.236.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } } }
I20260925 15:04:55.008199 30190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } } }
I20260925 15:04:55.008688 30190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:04:55.008989 30157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5cdd435e9804a6c820218e27b67d29d. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } } }
I20260925 15:04:55.009223 30157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:04:55.009076 30187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } } }
I20260925 15:04:55.009392 30187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:04:55.034178 30084 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 LEARNER]: Raft consensus shutting down.
I20260925 15:04:55.034720 30084 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 LEARNER]: Raft consensus is shut down!
I20260925 15:04:55.034885 30084 tablet_replica.cc:330] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: stopping tablet replica
I20260925 15:04:55.041858 30084 master.cc:552] Master@127.22.236.61:40973 shutdown complete.
I20260925 15:04:55.041921 30084 master_runner.cc:305] Clearing existing system tablet
I20260925 15:04:55.042207 30084 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:55.042266 30084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:55.044328 30084 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:04:55.045585 30192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:55.045835 30084 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:55.046095 30084 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:04:55.046177 30084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/wal
uuid: "69619aa463594ba086f18ec78c678cc4"
format_stamp: "Formatted at 2026-09-25 15:04:54 on dist-test-slave-nsrc"
server_key: "0518037be96ccfc32b68e11b4b2f553f"
server_key_iv: "733bc4f99cd95a48b6d9941fcce81060"
server_key_version: "encryptionkey@0"
I20260925 15:04:55.049037 30084 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:55.053277 30084 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 15:04:55.053411 30084 log.cc:1196] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 15:04:55.053807 30084 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: Deleting consensus metadata
I20260925 15:04:55.054011 30084 master_runner.cc:315] Copying system tablet from 127.22.236.62:45627
I20260925 15:04:55.056079 30084 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: tablet copy: Beginning tablet copy session from remote peer at address 127.22.236.62:45627
I20260925 15:04:55.067322 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.056745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45256 (local address 127.22.236.62:45627)
0925 15:04:55.056929 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:55.056936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:55.057369 (+ 433us) server_negotiation.cc:407] Connection header received
0925 15:04:55.057546 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:55.057552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:55.057631 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:55.057708 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:55.058447 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.059146 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.060460 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.060863 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.060968 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.062656 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:55.062660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:55.062664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:55.062692 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:55.065064 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.065607 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.065612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.065616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.065734 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.066059 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.066064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.066068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.066325 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:55.066456 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:55.066807 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 15:04:55.067003 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:04:55.067655 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.056703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:45256)
0925 15:04:55.057260 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:55.057283 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:55.057412 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:55.057750 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:55.057755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:55.058115 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:55.058308 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.058317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.059314 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.059318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.060325 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.060337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.061005 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.061010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.061191 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.061748 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:55.061764 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:55.062480 (+ 716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:55.065174 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.065179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.065184 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.065507 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.065830 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.065835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.065839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.065966 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.066428 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:55.066432 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:55.066580 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 15:04:55.066690 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:04:55.066987 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":339,"threads_started":1}
I20260925 15:04:55.068463 29643 tablet_copy_service.cc:140] P d5cdd435e9804a6c820218e27b67d29d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 69619aa463594ba086f18ec78c678cc4 ({username='oryx', principal='oryx/127.22.236.61@KRBTEST.COM'} at 127.0.0.1:45256)
I20260925 15:04:55.068693 29643 tablet_copy_service.cc:161] P d5cdd435e9804a6c820218e27b67d29d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 69619aa463594ba086f18ec78c678cc4 at {username='oryx', principal='oryx/127.22.236.61@KRBTEST.COM'} at 127.0.0.1:45256: session id = 69619aa463594ba086f18ec78c678cc4-00000000000000000000000000000000
I20260925 15:04:55.071540 29643 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:04:55.073267 30084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 15:04:55.077493 30084 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: tablet copy: Starting download of 0 data blocks...
I20260925 15:04:55.077740 30084 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: tablet copy: Starting download of 1 WAL segments...
I20260925 15:04:55.079741 30084 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:04:55.083297 29643 tablet_copy_service.cc:342] P d5cdd435e9804a6c820218e27b67d29d: Request end of tablet copy session 69619aa463594ba086f18ec78c678cc4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.236.61@KRBTEST.COM'} at 127.0.0.1:45256
I20260925 15:04:55.083457 29643 tablet_copy_service.cc:433] P d5cdd435e9804a6c820218e27b67d29d: ending tablet copy session 69619aa463594ba086f18ec78c678cc4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 69619aa463594ba086f18ec78c678cc4
I20260925 15:04:55.097711 30084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:55.102939 30199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:55.106063 30200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:55.110423 30202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:55.110826 30084 server_base.cc:1034] running on GCE node
I20260925 15:04:55.111114 30084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:55.111724 30084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:55.112908 30084 hybrid_clock.cc:648] HybridClock initialized: now 1790348695112850 us; error 33 us; skew 500 ppm
I20260925 15:04:55.113111 30084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:55 dist-test-slave-nsrc krb5kdc[29578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348695, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:55.114763 30084 init.cc:377] Logged in from keytab as oryx/127.22.236.61@KRBTEST.COM (short username oryx)
I20260925 15:04:55.115856 30084 webserver.cc:460] Webserver started at http://127.22.236.61:43643/ using document root <none> and password file <none>
I20260925 15:04:55.116196 30084 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:55.116274 30084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:55.118389 30084 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:04:55.119810 30209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:55.120249 30084 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:04:55.120350 30084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/wal
uuid: "69619aa463594ba086f18ec78c678cc4"
format_stamp: "Formatted at 2026-09-25 15:04:54 on dist-test-slave-nsrc"
server_key: "0518037be96ccfc32b68e11b4b2f553f"
server_key_iv: "733bc4f99cd95a48b6d9941fcce81060"
server_key_version: "encryptionkey@0"
I20260925 15:04:55.120500 30084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:55.143168 30084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:55.146310 30084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:55.170313 30084 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.61:40973
I20260925 15:04:55.171129 30084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-1/data/info.pb
I20260925 15:04:55.170313 30260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.61:40973 every 8 connection(s)
I20260925 15:04:55.176491 30261 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:04:55.179522 30261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: Bootstrap starting.
I20260925 15:04:55.189752 30261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:04:55.190158 30261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: Bootstrap complete.
I20260925 15:04:55.190920 30261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } }
I20260925 15:04:55.191103 30261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 69619aa463594ba086f18ec78c678cc4, State: Initialized, Role: LEARNER
I20260925 15:04:55.191349 30261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } }
I20260925 15:04:55.192047 30264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } } }
I20260925 15:04:55.192340 30261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:55.192510 30264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:04:55.204627 30275 catalog_manager.cc:1259] Loaded cluster ID: d8f4c53ce2424dabaec159f4fffbc069
I20260925 15:04:55.204756 30275 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: loading cluster ID for follower catalog manager: success
I20260925 15:04:55.208348 30275 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: acquiring CA information for follower catalog manager: success
I20260925 15:04:55.210701 30275 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 15:04:55.339920 30183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:55.340231 30183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:55.340371 30183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:55.347965 30183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:04:55.348052 30183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:55.348117 30183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:55.348214 30183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:04:55.348268 30183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:04:55.357805 30183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:45627,127.22.236.61:40973,127.22.236.60:38781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.60.keytab
--principal=oryx/127.22.236.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.60:38781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.236.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:55.358942 30183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:55.360117 30183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:55.370988 30281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:55.371016 30283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:55.370954 30280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:55.371922 30183 server_base.cc:1034] running on GCE node
I20260925 15:04:55.372710 30183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:55.374047 30183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:55.375238 30183 hybrid_clock.cc:648] HybridClock initialized: now 1790348695375188 us; error 45 us; skew 500 ppm
I20260925 15:04:55.375566 30183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:55 dist-test-slave-nsrc krb5kdc[29578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348695, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:55.378448 30183 init.cc:377] Logged in from keytab as oryx/127.22.236.60@KRBTEST.COM (short username oryx)
I20260925 15:04:55.379750 30183 webserver.cc:460] Webserver started at http://127.22.236.60:42889/ using document root <none> and password file <none>
I20260925 15:04:55.380211 30079 heartbeater.cc:503] Master 127.22.236.62:45627 was elected leader, sending a full tablet report...
I20260925 15:04:55.380445 30183 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:55.380573 30183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:55.380841 30183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:04:55.384182 30183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/data/instance:
uuid: "bf3d83eb24d040468c204d3217258138"
format_stamp: "Formatted at 2026-09-25 15:04:55 on dist-test-slave-nsrc"
server_key: "e7e389171165ba9dce329f2a9a39d83b"
server_key_iv: "808bb2a160eb0d4184a8d67fb1789ed5"
server_key_version: "encryptionkey@0"
I20260925 15:04:55.384964 30183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/wal/instance:
uuid: "bf3d83eb24d040468c204d3217258138"
format_stamp: "Formatted at 2026-09-25 15:04:55 on dist-test-slave-nsrc"
server_key: "e7e389171165ba9dce329f2a9a39d83b"
server_key_iv: "808bb2a160eb0d4184a8d67fb1789ed5"
server_key_version: "encryptionkey@0"
I20260925 15:04:55.390056 30183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:04:55.393352 30290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:55.394702 30183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:04:55.394848 30183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/wal
uuid: "bf3d83eb24d040468c204d3217258138"
format_stamp: "Formatted at 2026-09-25 15:04:55 on dist-test-slave-nsrc"
server_key: "e7e389171165ba9dce329f2a9a39d83b"
server_key_iv: "808bb2a160eb0d4184a8d67fb1789ed5"
server_key_version: "encryptionkey@0"
I20260925 15:04:55.395018 30183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:55.413921 30183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:55.418114 30183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:55.418341 30183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:55.435094 30183 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.60:38781
I20260925 15:04:55.435112 30341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.60:38781 every 8 connection(s)
I20260925 15:04:55.436369 30183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/data/info.pb
I20260925 15:04:55.440169 30342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:55.445750 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 30183
I20260925 15:04:55.446590 30342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:04:55 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348695, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.60@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
Sep 25 15:04:55 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.236.60@KRBTEST.COM
I20260925 15:04:55.464223 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.449752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45270 (local address 127.22.236.62:45627)
0925 15:04:55.449901 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:55.449907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:55.450438 (+ 531us) server_negotiation.cc:407] Connection header received
0925 15:04:55.450903 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:55.450909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:55.450988 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:55.451097 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:55.452300 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.453019 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.454982 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.455451 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.455542 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.458930 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:55.458936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:55.458939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:55.458969 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:55.461391 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.462182 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.462187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.462191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.462300 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.462657 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.462662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.462666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.462915 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:55.463053 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:55.463683 (+ 630us) server_negotiation.cc:299] Negotiation successful
0925 15:04:55.463889 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:04:55.465137 30346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.449653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:45270)
0925 15:04:55.450300 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:55.450341 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:55.450688 (+ 347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:55.451170 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:55.451176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:55.451794 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:55.452126 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.452171 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.453151 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.453157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.454853 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.454866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.455643 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.455650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.455869 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.456625 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:55.456644 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:55.458754 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:55.461502 (+ 2748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.461513 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.461525 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.462079 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.462393 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.462398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.462402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.462559 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.463029 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:55.463036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:55.463330 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 15:04:55.463578 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 15:04:55.464033 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":86,"thread_start_us":327,"threads_started":1}
I20260925 15:04:55.465154 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.447723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35730 (local address 127.22.236.60:38781)
0925 15:04:55.448421 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:55.448432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:55.448469 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:04:55.450517 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:55.450540 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:55.450818 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:55.450982 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:55.452330 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.453961 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.455152 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.455689 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.455870 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.459618 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:55.459625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:55.459636 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:55.459664 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:55.462155 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.462738 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.462744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.462746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.462853 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.463180 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.463185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.463188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.463495 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:55.463752 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:55.464073 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 15:04:55.464556 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":501,"thread_start_us":298,"threads_started":1}
I20260925 15:04:55.467017 30342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:04:55 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348695, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.60@KRBTEST.COM for oryx/127.22.236.61@KRBTEST.COM
I20260925 15:04:55.478626 30346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.467599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40973 (local address 127.0.0.1:35388)
0925 15:04:55.467755 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:55.467772 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:55.467848 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:55.468558 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:55.468561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:55.468846 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:55.468963 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.468970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.470083 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.470086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.470987 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.470998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.471626 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.471630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.471777 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.472229 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:55.472244 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:55.473924 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:55.476610 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.476615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.476619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.476943 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.477257 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.477262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.477266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.477380 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.477802 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:55.477806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:55.477935 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 15:04:55.478044 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 15:04:55.478299 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:04:55.478653 30347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.467778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.22.236.61:40973)
0925 15:04:55.468301 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:55.468307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:55.468324 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:04:55.468379 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:55.468383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:55.468451 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:55.468533 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:55.469069 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.469964 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.471083 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.471519 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.471618 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.474109 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:55.474114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:55.474117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:55.474146 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:55.476503 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.477047 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.477051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.477053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.477166 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.477467 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.477472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.477475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.477703 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:55.477828 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:55.478133 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 15:04:55.478307 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":232,"threads_started":1}
I20260925 15:04:55.479503 30342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:04:55 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348695, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.60@KRBTEST.COM for oryx/127.22.236.60@KRBTEST.COM
I20260925 15:04:55.490393 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.480179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35736 (local address 127.22.236.60:38781)
0925 15:04:55.480352 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:55.480358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:55.480405 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:04:55.480504 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:55.480508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:55.480556 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:55.480623 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:55.481152 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.481784 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.482812 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.483227 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.483314 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.486243 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:55.486248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:55.486251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:55.486277 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:55.488447 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.488934 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.488937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.488940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.489043 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.489340 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.489343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.489345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.489584 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:55.489682 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:55.490010 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 15:04:55.490156 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":74}
I20260925 15:04:55.490396 30346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.480195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:38781 (local address 127.0.0.1:35736)
0925 15:04:55.480342 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:55.480356 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:55.480432 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:55.480648 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:55.480651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:55.480932 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:55.481056 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.481063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.481883 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.481886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.482712 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.482722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.483328 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.483332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.483490 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.484328 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:55.484343 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:55.486063 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:55.488549 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.488552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.488554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.488860 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.489127 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.489130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.489133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.489262 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.489671 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:55.489674 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:55.489778 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 15:04:55.489886 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:04:55.490087 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 15:04:55.494867 30342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: Bootstrap starting.
I20260925 15:04:55.497797 30342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:55.498788 30342 log.cc:824] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:55.501296 30342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: No bootstrap required, opened a new log
I20260925 15:04:55.508268 30342 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } }
I20260925 15:04:55.508562 30342 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:55.508666 30342 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3d83eb24d040468c204d3217258138, State: Initialized, Role: FOLLOWER
I20260925 15:04:55.509239 30342 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } }
I20260925 15:04:55.509951 30349 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } } }
I20260925 15:04:55.510228 30349 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:04:55.510911 30342 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:04:55.519134 30360 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:04:55.519271 30360 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:04:55.520049 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 29684
I20260925 15:04:55.523492 30183 master_runner.cc:428] Detected that this master bf3d83eb24d040468c204d3217258138 is joining an existing cluster
I20260925 15:04:55.523602 30183 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.236.60:38781
Sep 25 15:04:55 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.236.1@KRBTEST.COM
I20260925 15:04:55.541021 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.521142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.22.236.1:46247)
0925 15:04:55.521754 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:55.521762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:55.521789 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:04:55.521853 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:55.521861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:55.522119 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:55.522274 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:55.523810 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.525456 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.527154 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.528007 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.528136 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.533823 (+ 5687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:55.533840 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:55.533859 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:55.533902 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:55.537962 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.538705 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.538714 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.538717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.538860 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.539200 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.539205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.539209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.539622 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:55.539895 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:55.540225 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 15:04:55.540646 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":287,"threads_started":1}
I20260925 15:04:55.542374 29807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:55.554486 30367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.537287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:38781 (local address 127.0.0.1:35738)
0925 15:04:55.538067 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:55.538086 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:55.538210 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:55.538523 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:55.538526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:55.540732 (+ 2206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:55.541277 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.541289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.542682 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.542686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.544684 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.544698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.545888 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.545895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.546057 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.546702 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:55.546723 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:55.547762 (+ 1039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:55.551172 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.551178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.551183 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.551677 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.552574 (+ 897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.552579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.552586 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.552736 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.553421 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:55.553426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:55.553594 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 15:04:55.553728 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 15:04:55.554039 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":439,"threads_started":1}
I20260925 15:04:55.558413 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.537216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.22.236.60:38781)
0925 15:04:55.537403 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:55.537413 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:55.538148 (+ 735us) server_negotiation.cc:407] Connection header received
0925 15:04:55.538322 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:55.538327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:55.538412 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:55.538507 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:55.541428 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.542461 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.544845 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.545535 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.545632 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.547938 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:55.547944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:55.547949 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:55.547980 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:55.551050 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.551805 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.551810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.551813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.552456 (+ 643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.552841 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.552846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.552849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.553218 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:55.553368 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:55.557657 (+ 4289us) server_negotiation.cc:299] Negotiation successful
0925 15:04:55.558015 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:04:55.562290 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.537671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45284 (local address 127.22.236.62:45627)
0925 15:04:55.537865 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:55.537871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:55.541863 (+ 3992us) server_negotiation.cc:407] Connection header received
0925 15:04:55.542082 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:55.542088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:55.542162 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:55.542225 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:55.544430 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.545489 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.547504 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.548090 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.548222 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.554244 (+ 6022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:55.554252 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:55.554256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:55.554295 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:55.557533 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.560016 (+ 2483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.560021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.560024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.560210 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.560633 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.560639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.560642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.560956 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:55.561086 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:55.561743 (+ 657us) server_negotiation.cc:299] Negotiation successful
0925 15:04:55.561961 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:04:55.563443 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.539532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:45284)
0925 15:04:55.541763 (+ 2231us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:55.541784 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:55.541951 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:55.542409 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:55.542413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:55.543754 (+ 1341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:55.544091 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.544103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.545694 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.545698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.547231 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.547244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.548504 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.548510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.548659 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.549740 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:55.549760 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:55.552802 (+ 3042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:55.558841 (+ 6039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.558847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.558850 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.559201 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.560333 (+ 1132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.560337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.560341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.560519 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.561153 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:55.561158 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:55.561407 (+ 249us) client_negotiation.cc:769] Sending connection context
0925 15:04:55.562683 (+ 1276us) client_negotiation.cc:240] Negotiation successful
0925 15:04:55.563011 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2110,"thread_start_us":10491,"threads_started":1}
I20260925 15:04:55.574064 30347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.538284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.22.236.61:40973)
0925 15:04:55.538431 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:55.538436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:55.556672 (+ 18236us) server_negotiation.cc:407] Connection header received
0925 15:04:55.557192 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:55.557198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:55.557288 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:55.557366 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:55.559928 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.561030 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.563413 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.564074 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.564193 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.566997 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:55.567004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:55.567008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:55.567047 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:55.570802 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.571527 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.571532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.571535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.571676 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.572478 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.572483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.572487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.572831 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:55.572954 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:55.573530 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 15:04:55.573749 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:04:55.575356 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.540846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40973 (local address 127.0.0.1:35404)
0925 15:04:55.556569 (+ 15723us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:55.556589 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:55.557028 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:55.557551 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:55.557556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:55.559470 (+ 1914us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:55.559774 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.559786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.561196 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.561201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.563255 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.563269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.564959 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.564965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.565136 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.565739 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:55.565760 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:55.566796 (+ 1036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:55.570965 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.570970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.570974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.571409 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.571836 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.571841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.571845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.572324 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.573043 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:55.573048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:55.573229 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 15:04:55.574294 (+ 1065us) client_negotiation.cc:240] Negotiation successful
0925 15:04:55.574919 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15576,"thread_start_us":15858,"threads_started":1}
Sep 25 15:04:55 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348692, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.62@KRBTEST.COM for oryx/127.22.236.60@KRBTEST.COM
I20260925 15:04:55.588124 30371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.571779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:38781 (local address 127.0.0.1:35744)
0925 15:04:55.572281 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:55.572300 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:55.572473 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:55.573008 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:55.573013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:55.573447 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:55.573656 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.573667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.575831 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.575836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.577299 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.577312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.578282 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.578288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.578473 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.579044 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:55.579065 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:55.581680 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:55.585475 (+ 3795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.585481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.585486 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.586012 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.586380 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.586385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.586389 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.586532 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.587182 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:55.587187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:55.587327 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 15:04:55.587465 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 15:04:55.587720 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":278,"threads_started":1}
I20260925 15:04:55.588889 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.572103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.22.236.60:38781)
0925 15:04:55.572294 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:55.572299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:55.572387 (+ 88us) server_negotiation.cc:407] Connection header received
0925 15:04:55.572632 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:55.572638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:55.572734 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:55.572796 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:55.573822 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.575661 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.577459 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.578133 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.578257 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.581859 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:55.581866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:55.581871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:55.581904 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:55.585349 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.586133 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.586139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.586142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.586272 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.586651 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.586656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.586660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.587074 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:55.587216 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:55.588342 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0925 15:04:55.588565 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 15:04:55.590533 29622 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.236.60:38781: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bf3d83eb24d040468c204d3217258138"
member_type: NON_VOTER
last_known_addr {
host: "127.22.236.60"
port: 38781
}
attrs {
promote: true
}
}
dest_uuid: "d5cdd435e9804a6c820218e27b67d29d"
cas_config_opid_index: 5
I20260925 15:04:55.591557 29622 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } }
I20260925 15:04:55.594195 30317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:55.598019 30317 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 1 FOLLOWER]: Refusing update from remote peer d5cdd435e9804a6c820218e27b67d29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 15:04:55.598362 30373 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bf3d83eb24d040468c204d3217258138 (127.22.236.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } } }
I20260925 15:04:55.605726 30374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:55.611615 30373 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.236.60:38781
I20260925 15:04:55.612108 30183 master.cc:537] Master@127.22.236.60:38781 shutting down...
I20260925 15:04:55.613081 30347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.605661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.22.236.61:40973)
0925 15:04:55.605823 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:55.605829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:55.606205 (+ 376us) server_negotiation.cc:407] Connection header received
0925 15:04:55.606273 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:55.606279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:55.606368 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:55.606469 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:55.606917 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.607939 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.610357 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.611566 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.611672 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.611928 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:55.612693 (+ 765us) server_negotiation.cc:299] Negotiation successful
0925 15:04:55.612832 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:04:55.613549 30371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.605336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40973 (local address 127.0.0.1:35414)
0925 15:04:55.605952 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:55.605970 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:55.606076 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:55.606528 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:55.606533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:55.606562 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:55.606744 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.606755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.608114 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.608120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.610178 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.610192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.612103 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.612109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.612359 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.612431 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 15:04:55.612520 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:04:55.613292 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509}
I20260925 15:04:55.614518 30236 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 LEARNER]: Refusing update from remote peer d5cdd435e9804a6c820218e27b67d29d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260925 15:04:55.615900 29608 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 15:04:55.625473 30374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 15:04:55.626078 30375 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } } }
I20260925 15:04:55.633267 30375 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: This master's current role is: LEADER
W20260925 15:04:55.633981 29608 consensus_peers.cc:487] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d -> Peer bf3d83eb24d040468c204d3217258138 (127.22.236.60:38781): Couldn't send request to peer bf3d83eb24d040468c204d3217258138. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 15:04:55.639194 30236 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bf3d83eb24d040468c204d3217258138 (127.22.236.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } } }
I20260925 15:04:55.643527 30264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5cdd435e9804a6c820218e27b67d29d. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } } }
I20260925 15:04:55.643811 30264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:04:55.643455 30379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: NON_VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: true } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } } }
I20260925 15:04:55.644030 30379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:04:55.644697 30375 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d: attempting to promote NON_VOTER 69619aa463594ba086f18ec78c678cc4 to VOTER
I20260925 15:04:55.645570 30375 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } }
W20260925 15:04:55.647900 29608 consensus_peers.cc:487] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d -> Peer bf3d83eb24d040468c204d3217258138 (127.22.236.60:38781): Couldn't send request to peer bf3d83eb24d040468c204d3217258138. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 15:04:55.648074 30236 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 LEARNER]: Refusing update from remote peer d5cdd435e9804a6c820218e27b67d29d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 15:04:55.648676 30375 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 15:04:55.650462 30183 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:55.650854 30183 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:55.651060 30183 tablet_replica.cc:330] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: stopping tablet replica
I20260925 15:04:55.652339 30373 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 69619aa463594ba086f18ec78c678cc4 (127.22.236.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } } }
I20260925 15:04:55.654014 30236 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 69619aa463594ba086f18ec78c678cc4 (127.22.236.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } } }
I20260925 15:04:55.655346 30373 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } } }
I20260925 15:04:55.656124 30373 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:55.655571 30372 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } } }
I20260925 15:04:55.656383 30372 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:55.658448 30379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } } }
I20260925 15:04:55.658730 30379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:04:55.659686 30183 master.cc:552] Master@127.22.236.60:38781 shutdown complete.
I20260925 15:04:55.659765 30183 master_runner.cc:305] Clearing existing system tablet
I20260925 15:04:55.660135 30183 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:55.660265 30183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:55.662976 30183 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:04:55.664462 30381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:55.664834 30183 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:55.665082 30183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:04:55.665171 30183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/wal
uuid: "bf3d83eb24d040468c204d3217258138"
format_stamp: "Formatted at 2026-09-25 15:04:55 on dist-test-slave-nsrc"
server_key: "e7e389171165ba9dce329f2a9a39d83b"
server_key_iv: "808bb2a160eb0d4184a8d67fb1789ed5"
server_key_version: "encryptionkey@0"
I20260925 15:04:55.669885 30183 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:55.676421 30183 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 15:04:55.676559 30183 log.cc:1196] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 15:04:55.677278 30183 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: Deleting consensus metadata
I20260925 15:04:55.677515 30183 master_runner.cc:315] Copying system tablet from 127.22.236.62:45627
I20260925 15:04:55.680827 30183 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: tablet copy: Beginning tablet copy session from remote peer at address 127.22.236.62:45627
I20260925 15:04:55.687987 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 29684
I20260925 15:04:55.696646 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.681991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45290 (local address 127.22.236.62:45627)
0925 15:04:55.682206 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:55.682212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:55.682235 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:04:55.682474 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:55.682481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:55.682561 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:55.682665 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:55.683622 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.684841 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.686841 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.687524 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.687618 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.690334 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:55.690341 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:55.690347 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:55.690394 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:55.693760 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.694530 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.694535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.694539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.694678 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:55.695079 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:55.695084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:55.695087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:55.695428 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:55.695576 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:55.696001 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 15:04:55.696267 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 15:04:55.696652 30387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:55.681586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:45290)
0925 15:04:55.682142 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:55.682162 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:55.682301 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:55.682710 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:55.682715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:55.683202 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:55.683457 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.683471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.685010 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.685016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.686685 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:55.686701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:55.687885 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:55.687891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:55.688074 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:55.688904 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:55.688927 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:55.690095 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:55.693914 (+ 3819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.693920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.693924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.694408 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.694790 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:55.694794 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:55.694798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:55.694975 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:55.695548 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:55.695553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:55.695740 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 15:04:55.695890 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 15:04:55.696247 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":653,"threads_started":1}
I20260925 15:04:55.697257 29643 tablet_copy_service.cc:140] P d5cdd435e9804a6c820218e27b67d29d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bf3d83eb24d040468c204d3217258138 ({username='oryx', principal='oryx/127.22.236.60@KRBTEST.COM'} at 127.0.0.1:45290)
I20260925 15:04:55.697417 29643 tablet_copy_service.cc:161] P d5cdd435e9804a6c820218e27b67d29d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bf3d83eb24d040468c204d3217258138 at {username='oryx', principal='oryx/127.22.236.60@KRBTEST.COM'} at 127.0.0.1:45290: session id = bf3d83eb24d040468c204d3217258138-00000000000000000000000000000000
I20260925 15:04:55.700989 29643 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:04:55.702915 30183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 15:04:55.705354 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:46247
--local_ip_for_outbound_sockets=127.22.236.1
--tserver_master_addrs=127.22.236.62:45627
--webserver_port=39043
--webserver_interface=127.22.236.1
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.1.keytab
--principal=oryx/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.236.62:45627,127.22.236.61:40973
--tserver_master_addrs=127.22.236.62:45627,127.22.236.61:40973,127.22.236.60:38781 with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:04:55.709106 30183 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: tablet copy: Starting download of 0 data blocks...
I20260925 15:04:55.709362 30183 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: tablet copy: Starting download of 1 WAL segments...
I20260925 15:04:55.711867 30183 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:04:55.715979 29643 tablet_copy_service.cc:342] P d5cdd435e9804a6c820218e27b67d29d: Request end of tablet copy session bf3d83eb24d040468c204d3217258138-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.236.60@KRBTEST.COM'} at 127.0.0.1:45290
I20260925 15:04:55.716112 29643 tablet_copy_service.cc:433] P d5cdd435e9804a6c820218e27b67d29d: ending tablet copy session bf3d83eb24d040468c204d3217258138-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bf3d83eb24d040468c204d3217258138
I20260925 15:04:55.727283 30183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:55.730069 30389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:55.730448 30390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:55.737672 30392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:55.740856 30183 server_base.cc:1034] running on GCE node
I20260925 15:04:55.741214 30183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:55.741837 30183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:55.743032 30183 hybrid_clock.cc:648] HybridClock initialized: now 1790348695742987 us; error 32 us; skew 500 ppm
I20260925 15:04:55.743238 30183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:55 dist-test-slave-nsrc krb5kdc[29578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348695, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:55.745496 30183 init.cc:377] Logged in from keytab as oryx/127.22.236.60@KRBTEST.COM (short username oryx)
I20260925 15:04:55.747282 30183 webserver.cc:460] Webserver started at http://127.22.236.60:45833/ using document root <none> and password file <none>
I20260925 15:04:55.747592 30183 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:55.747689 30183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:55.750662 30183 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260925 15:04:55.752530 30399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:55.752936 30183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 15:04:55.753054 30183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/wal
uuid: "bf3d83eb24d040468c204d3217258138"
format_stamp: "Formatted at 2026-09-25 15:04:55 on dist-test-slave-nsrc"
server_key: "e7e389171165ba9dce329f2a9a39d83b"
server_key_iv: "808bb2a160eb0d4184a8d67fb1789ed5"
server_key_version: "encryptionkey@0"
I20260925 15:04:55.753249 30183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:55.780510 30183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:55.784461 30183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:55.807551 30183 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.60:38781
I20260925 15:04:55.807616 30450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.60:38781 every 8 connection(s)
I20260925 15:04:55.808552 30183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/master-2/data/info.pb
I20260925 15:04:55.813616 30451 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:04:55.817502 30451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: Bootstrap starting.
I20260925 15:04:55.835660 30451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:04:55.836210 30451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: Bootstrap complete.
I20260925 15:04:55.837230 30451 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } }
I20260925 15:04:55.837538 30451 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf3d83eb24d040468c204d3217258138, State: Initialized, Role: LEARNER
I20260925 15:04:55.837796 30451 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } }
I20260925 15:04:55.838634 30454 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } } }
I20260925 15:04:55.839035 30454 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:04:55.838894 30451 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:55.845911 30465 catalog_manager.cc:1259] Loaded cluster ID: d8f4c53ce2424dabaec159f4fffbc069
I20260925 15:04:55.850613 30465 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: loading cluster ID for follower catalog manager: success
I20260925 15:04:55.854225 30465 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: acquiring CA information for follower catalog manager: success
I20260925 15:04:55.857004 30465 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 15:04:56.053256 30388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:56.053547 30388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:56.053687 30388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:56.061112 30388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:56.061228 30388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:56.061369 30388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:04:56.070784 30388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.1.keytab
--principal=oryx/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:46247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=39043
--tserver_master_addrs=127.22.236.62:45627,127.22.236.61:40973,127.22.236.60:38781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:56.071992 30388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:56.073253 30388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:56.086423 30470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:56.086421 30471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:56.086696 30473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:56.087234 30388 server_base.cc:1034] running on GCE node
I20260925 15:04:56.087736 30388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:56.088598 30388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:56.089774 30388 hybrid_clock.cc:648] HybridClock initialized: now 1790348696089734 us; error 34 us; skew 500 ppm
I20260925 15:04:56.090050 30388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:56 dist-test-slave-nsrc krb5kdc[29578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348696, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:56.092887 30388 init.cc:377] Logged in from keytab as oryx/127.22.236.1@KRBTEST.COM (short username oryx)
I20260925 15:04:56.094198 30388 webserver.cc:460] Webserver started at http://127.22.236.1:39043/ using document root <none> and password file <none>
I20260925 15:04:56.094803 30388 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:56.095290 30388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:56.100401 30388 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:04:56.102888 30480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:56.104221 30388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:56.104343 30388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "3ba42b62df1045df88a14643e5ac471a"
format_stamp: "Formatted at 2026-09-25 15:04:53 on dist-test-slave-nsrc"
server_key: "0341972dcd22775ed09f123281d3e031"
server_key_iv: "06e549948ac898e77079795d59996896"
server_key_version: "encryptionkey@0"
I20260925 15:04:56.105206 30388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:56.122720 30485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.117166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:38781 (local address 127.0.0.1:35760)
0925 15:04:56.117548 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:56.117570 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:56.117718 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:56.118105 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:56.118110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:56.118137 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:56.118294 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:56.118303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.119477 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.119481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:56.120707 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:56.120720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.121958 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.121964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:56.122147 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.122178 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:04:56.122258 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 15:04:56.122389 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":195,"threads_started":1}
I20260925 15:04:56.122720 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.117259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.22.236.60:38781)
0925 15:04:56.117813 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:56.117819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:56.117837 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:04:56.117896 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:56.117899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:56.117971 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:56.118065 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:56.118415 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.119336 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.120836 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.121801 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.121930 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.122126 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:56.122290 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 15:04:56.122412 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":329,"threads_started":1}
I20260925 15:04:56.123600 30426 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 15:04:56.124220 30388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:56.124059 30454 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5cdd435e9804a6c820218e27b67d29d. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } } }
I20260925 15:04:56.124337 30454 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:04:56.128232 30388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:56.128454 30388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:56.129307 30388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:56.130483 30388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:56.130578 30388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:56.130681 30388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:56.130744 30388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:56.160065 30388 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:46247
I20260925 15:04:56.160116 30594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:46247 every 8 connection(s)
I20260925 15:04:56.161392 30388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:04:56.167716 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 30388
I20260925 15:04:56.167990 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 29818
Sep 25 15:04:56 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348696, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
Sep 25 15:04:56 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.236.2@KRBTEST.COM
Sep 25 15:04:56 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348696, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.60@KRBTEST.COM
Sep 25 15:04:56 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348696, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.61@KRBTEST.COM
I20260925 15:04:56.190289 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.164774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:55615 (local address 127.22.236.62:45627)
0925 15:04:56.164972 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:56.164980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:56.166305 (+ 1325us) server_negotiation.cc:407] Connection header received
0925 15:04:56.168544 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:56.168550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:56.168630 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:56.168696 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:56.172791 (+ 4095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.173808 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.176298 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.176862 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.176965 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.181959 (+ 4994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:56.181967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:56.181971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:56.182011 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:56.185317 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.187024 (+ 1707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.187029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.187033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.187177 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.187693 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.187698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.187701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.188096 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:56.188303 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:56.189715 (+ 1412us) server_negotiation.cc:299] Negotiation successful
0925 15:04:56.189947 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:04:56.191736 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.165619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:57631 (local address 127.22.236.60:38781)
0925 15:04:56.165789 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:56.165795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:56.167155 (+ 1360us) server_negotiation.cc:407] Connection header received
0925 15:04:56.168526 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:56.168532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:56.168608 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:56.168675 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:56.172659 (+ 3984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.173711 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.176286 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.176880 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.176960 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.185367 (+ 8407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:56.185373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:56.185378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:56.185418 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:56.188932 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.189782 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.189788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.189792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.189920 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.190297 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.190302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.190305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.190634 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:56.190756 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:56.191204 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 15:04:56.191429 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:04:56.192175 30602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.165335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.1:55615)
0925 15:04:56.166193 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:56.166210 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:56.168237 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:56.169165 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:56.169176 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:56.170984 (+ 1808us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:56.172653 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:56.172663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.173986 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.173994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:56.176121 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:56.176135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.177274 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.177280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:56.177609 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.178564 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:56.178602 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:56.181683 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:56.186271 (+ 4588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:56.186284 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:56.186302 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:56.186876 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:56.187329 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:56.187334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:56.187337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:56.187564 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:56.188374 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:56.188383 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:56.188782 (+ 399us) client_negotiation.cc:769] Sending connection context
0925 15:04:56.189096 (+ 314us) client_negotiation.cc:240] Negotiation successful
0925 15:04:56.190591 (+ 1495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":172,"spinlock_wait_cycles":42496,"thread_start_us":406,"threads_started":1}
I20260925 15:04:56.192606 30601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.165224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:38781 (local address 127.22.236.1:57631)
0925 15:04:56.167057 (+ 1833us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:56.167069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:56.168226 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:56.169165 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:56.169176 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:56.171162 (+ 1986us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:56.172470 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:56.172497 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.173881 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.173887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:56.176126 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:56.176157 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.177273 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.177279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:56.177614 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.182717 (+ 5103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:56.182739 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:56.185160 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:56.189078 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:56.189084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:56.189088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:56.189652 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:56.190052 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:56.190056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:56.190059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:56.190188 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:56.190828 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:56.190833 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:56.190969 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 15:04:56.191897 (+ 928us) client_negotiation.cc:240] Negotiation successful
0925 15:04:56.192186 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1715,"mutex_wait_us":621,"spinlock_wait_cycles":3840,"thread_start_us":969,"threads_started":1}
I20260925 15:04:56.193848 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.169076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.22.236.2:38433)
0925 15:04:56.169953 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:56.169962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:56.170462 (+ 500us) server_negotiation.cc:407] Connection header received
0925 15:04:56.170737 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:56.170746 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:56.170973 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:56.171107 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:56.173318 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.174706 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.176471 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.177246 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.177334 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.185919 (+ 8585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:56.185933 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:56.185952 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:56.185991 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:56.190450 (+ 4459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.191270 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.191278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.191281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.191430 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.191856 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.191862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.191865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.192287 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:56.192560 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:56.193245 (+ 685us) server_negotiation.cc:299] Negotiation successful
0925 15:04:56.193497 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":583,"threads_started":1}
I20260925 15:04:56.193955 30596 heartbeater.cc:347] Connected to a master server at 127.22.236.62:45627
I20260925 15:04:56.194248 30596 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:56.194298 30599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.164316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:42611 (local address 127.22.236.61:40973)
0925 15:04:56.165503 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:56.165517 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:56.166310 (+ 793us) server_negotiation.cc:407] Connection header received
0925 15:04:56.168533 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:56.168538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:56.168608 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:56.168676 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:56.178094 (+ 9418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.179224 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.181168 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.181806 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.181903 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.186477 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:56.186483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:56.186487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:56.186526 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:56.190045 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.190996 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.191001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.191005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.191145 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.191558 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.191563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.191567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.191863 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:56.192024 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:56.193793 (+ 1769us) server_negotiation.cc:299] Negotiation successful
0925 15:04:56.193983 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":973,"threads_started":1}
I20260925 15:04:56.194859 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.165224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40973 (local address 127.22.236.1:42611)
0925 15:04:56.166131 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:56.166193 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:56.168226 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:56.169165 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:56.169176 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:56.171040 (+ 1864us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:56.177192 (+ 6152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:56.177647 (+ 455us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.179371 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.179375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:56.181019 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:56.181032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.182215 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.182221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:56.182384 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.183767 (+ 1383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:56.183788 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:56.186253 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:56.190211 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:56.190216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:56.190219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:56.190881 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:56.191268 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:56.191272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:56.191275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:56.191440 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:56.192500 (+ 1060us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:56.192504 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:56.193552 (+ 1048us) client_negotiation.cc:769] Sending connection context
0925 15:04:56.193673 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 15:04:56.194196 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"spinlock_wait_cycles":1280,"thread_start_us":1459,"threads_started":1}
I20260925 15:04:56.195137 30596 heartbeater.cc:511] Master 127.22.236.62:45627 requested a full tablet report, sending...
I20260925 15:04:56.195632 30595 heartbeater.cc:347] Connected to a master server at 127.22.236.60:38781
I20260925 15:04:56.195678 30597 heartbeater.cc:347] Connected to a master server at 127.22.236.61:40973
I20260925 15:04:56.195731 29941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:56.195760 30595 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:56.196060 30597 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:56.196532 30595 heartbeater.cc:511] Master 127.22.236.60:38781 requested a full tablet report, sending...
I20260925 15:04:56.196801 29622 ts_manager.cc:194] Re-registered known tserver with Master: 3ba42b62df1045df88a14643e5ac471a (127.22.236.1:46247)
I20260925 15:04:56.197496 30597 heartbeater.cc:511] Master 127.22.236.61:40973 requested a full tablet report, sending...
I20260925 15:04:56.197853 29622 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:55615
I20260925 15:04:56.198347 30416 ts_manager.cc:194] Registered new tserver with Master: 3ba42b62df1045df88a14643e5ac471a (127.22.236.1:46247)
I20260925 15:04:56.199126 30226 ts_manager.cc:194] Registered new tserver with Master: 3ba42b62df1045df88a14643e5ac471a (127.22.236.1:46247)
I20260925 15:04:56.276683 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 29818
I20260925 15:04:56.289264 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:38433
--local_ip_for_outbound_sockets=127.22.236.2
--tserver_master_addrs=127.22.236.62:45627
--webserver_port=36319
--webserver_interface=127.22.236.2
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.2.keytab
--principal=oryx/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.236.62:45627,127.22.236.61:40973
--tserver_master_addrs=127.22.236.62:45627,127.22.236.61:40973,127.22.236.60:38781 with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:04:56.532987 30375 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: NON_VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: true } } }
I20260925 15:04:56.533349 30375 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:56.534860 30607 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d: attempting to promote NON_VOTER bf3d83eb24d040468c204d3217258138 to VOTER
I20260925 15:04:56.535388 30607 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } }
I20260925 15:04:56.537065 30426 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 1 LEARNER]: Refusing update from remote peer d5cdd435e9804a6c820218e27b67d29d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 15:04:56.537115 30236 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 FOLLOWER]: Refusing update from remote peer d5cdd435e9804a6c820218e27b67d29d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 15:04:56.537699 30375 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 15:04:56.538029 30607 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 15:04:56.540259 30607 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bf3d83eb24d040468c204d3217258138 (127.22.236.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } } }
I20260925 15:04:56.540967 30236 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bf3d83eb24d040468c204d3217258138 (127.22.236.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } } }
I20260925 15:04:56.550200 30607 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } } }
I20260925 15:04:56.550669 30375 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } } }
I20260925 15:04:56.550961 30375 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:56.551162 30607 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:56.552294 30611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } } }
I20260925 15:04:56.552696 30611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:04:56.553130 30608 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } } }
I20260925 15:04:56.553445 30608 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:56.553334 30426 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bf3d83eb24d040468c204d3217258138 (127.22.236.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } } }
I20260925 15:04:56.556423 30454 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d5cdd435e9804a6c820218e27b67d29d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } } }
I20260925 15:04:56.556665 30454 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:04:56.598521 30606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:56.598778 30606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:56.598909 30606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:56.606143 30606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:56.606242 30606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:56.606387 30606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:04:56.615777 30606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.2.keytab
--principal=oryx/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:38433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=36319
--tserver_master_addrs=127.22.236.62:45627,127.22.236.61:40973,127.22.236.60:38781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:56.616984 30606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:56.618257 30606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:56.630272 30617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:56.630316 30618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:56.630343 30620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:56.631300 30606 server_base.cc:1034] running on GCE node
I20260925 15:04:56.631819 30606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:56.632696 30606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:56.633876 30606 hybrid_clock.cc:648] HybridClock initialized: now 1790348696633842 us; error 67 us; skew 500 ppm
I20260925 15:04:56.634142 30606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:56 dist-test-slave-nsrc krb5kdc[29578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348696, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:56.637061 30606 init.cc:377] Logged in from keytab as oryx/127.22.236.2@KRBTEST.COM (short username oryx)
I20260925 15:04:56.638357 30606 webserver.cc:460] Webserver started at http://127.22.236.2:36319/ using document root <none> and password file <none>
I20260925 15:04:56.638971 30606 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:56.639087 30606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:56.644274 30606 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:04:56.646746 30627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:56.648041 30606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:04:56.648207 30606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "0b635f83f59d4b2b9f169b982f49c3f5"
format_stamp: "Formatted at 2026-09-25 15:04:53 on dist-test-slave-nsrc"
server_key: "70f03adf0692a58b21fc386d24fb9780"
server_key_iv: "d619e49fff9331903adb1934fe7d7ba5"
server_key_version: "encryptionkey@0"
I20260925 15:04:56.649075 30606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:56.664410 30606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:56.668234 30606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:56.668449 30606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:56.669297 30606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:56.670516 30606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:56.670598 30606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:56.670673 30606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:56.670723 30606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:56.700796 30606 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:38433
I20260925 15:04:56.700829 30739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:38433 every 8 connection(s)
I20260925 15:04:56.702186 30606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:04:56.706710 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 30606
I20260925 15:04:56.706996 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 29951
Sep 25 15:04:56 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348696, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.60@KRBTEST.COM
Sep 25 15:04:56 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348696, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
Sep 25 15:04:56 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.236.3@KRBTEST.COM
Sep 25 15:04:56 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348696, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.61@KRBTEST.COM
I20260925 15:04:56.729945 30744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.704947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:44415 (local address 127.22.236.60:38781)
0925 15:04:56.705453 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:56.705459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:56.710229 (+ 4770us) server_negotiation.cc:407] Connection header received
0925 15:04:56.710475 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:56.710481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:56.710562 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:56.710669 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:56.713086 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.714279 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.716857 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.717506 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.717607 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.723299 (+ 5692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:56.723303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:56.723306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:56.723343 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:56.726399 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.727844 (+ 1445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.727849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.727852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.727994 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.728415 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.728420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.728423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.728704 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:56.728818 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:56.729366 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 15:04:56.729556 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":240,"threads_started":1}
I20260925 15:04:56.729943 30747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.706077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:40485 (local address 127.22.236.62:45627)
0925 15:04:56.709723 (+ 3646us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:56.709730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:56.712352 (+ 2622us) server_negotiation.cc:407] Connection header received
0925 15:04:56.712602 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:56.712607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:56.712742 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:56.712845 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:56.713989 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.715047 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.716969 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.717585 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.717728 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.723248 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:56.723255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:56.723260 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:56.723299 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:56.726504 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.727323 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.727328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.727332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.727463 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.727967 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.727972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.727975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.728327 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:56.728442 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:56.729349 (+ 907us) server_negotiation.cc:299] Negotiation successful
0925 15:04:56.729558 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3536,"thread_start_us":3491,"threads_started":1}
I20260925 15:04:56.731799 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.708385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.22.236.3:35359)
0925 15:04:56.711303 (+ 2918us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:56.711309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:56.711331 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:04:56.711388 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:56.711393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:56.711554 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:56.711691 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:56.713653 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.715184 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.716969 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.717748 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.717850 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.723846 (+ 5996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:56.723872 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:56.723889 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:56.723930 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:56.727906 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.728730 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.728741 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.728744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.728899 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.729333 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.729338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.729342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.729727 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:56.730026 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:56.731194 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0925 15:04:56.731422 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2799,"thread_start_us":2682,"threads_started":1}
I20260925 15:04:56.732105 30750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.707724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.2:40485)
0925 15:04:56.712260 (+ 4536us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:56.712276 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:56.712453 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:56.713041 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:56.713045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:56.713554 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:56.713831 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:56.713843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.715218 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.715223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:56.716810 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:56.716824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.718005 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.718012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:56.718720 (+ 708us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.720526 (+ 1806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:56.720547 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:56.723074 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:56.726628 (+ 3554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:56.726632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:56.726636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:56.727197 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:56.727611 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:56.727615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:56.727619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:56.727857 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:56.728521 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:56.728528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:56.728930 (+ 402us) client_negotiation.cc:769] Sending connection context
0925 15:04:56.730212 (+ 1282us) client_negotiation.cc:240] Negotiation successful
0925 15:04:56.730719 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4426,"mutex_wait_us":502,"thread_start_us":4257,"threads_started":1}
I20260925 15:04:56.733196 30074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:56.733249 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.706414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:38781 (local address 127.22.236.2:44415)
0925 15:04:56.710140 (+ 3726us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:56.710157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:56.710321 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:56.711092 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:56.711102 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:56.711977 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:56.712883 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:56.712908 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.714481 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.714486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:56.716681 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:56.716700 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.718005 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.718012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:56.718276 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.719083 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:56.719111 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:56.722419 (+ 3308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:56.726558 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:56.726567 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:56.726584 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:56.727715 (+ 1131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:56.728123 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:56.728128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:56.728131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:56.728288 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:56.728888 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:56.728893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:56.729011 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 15:04:56.730514 (+ 1503us) client_negotiation.cc:240] Negotiation successful
0925 15:04:56.730830 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3617,"mutex_wait_us":96,"spinlock_wait_cycles":19712,"thread_start_us":3461,"threads_started":1}
I20260925 15:04:56.734000 30745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.705500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40973 (local address 127.22.236.2:45355)
0925 15:04:56.706286 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:56.706346 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:56.708988 (+ 2642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:56.718658 (+ 9670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:56.718663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:56.719066 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:56.719325 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:56.719338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.721421 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.721427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:56.722627 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:56.722638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.724368 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.724374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:56.724539 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.724979 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:56.725000 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:56.727934 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:56.731429 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:56.731434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:56.731437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:56.731895 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:56.732323 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:56.732327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:56.732331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:56.732529 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:56.733083 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:56.733088 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:56.733229 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 15:04:56.733379 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 15:04:56.733655 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":4977,"threads_started":1}
I20260925 15:04:56.734447 30746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:56.705990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:45355 (local address 127.22.236.61:40973)
0925 15:04:56.718332 (+ 12342us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:56.718337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:56.718355 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:04:56.718421 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:56.718426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:56.718518 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:56.718615 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:56.719494 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.721257 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.722754 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:56.724123 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:56.724769 (+ 646us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:56.728115 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:56.728121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:56.728126 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:56.728204 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:56.731282 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.732043 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.732047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.732050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.732198 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:56.732645 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:56.732648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:56.732650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:56.732928 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:56.733054 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:56.733812 (+ 758us) server_negotiation.cc:299] Negotiation successful
0925 15:04:56.734042 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12229,"thread_start_us":12129,"threads_started":1}
I20260925 15:04:56.734508 30740 heartbeater.cc:347] Connected to a master server at 127.22.236.60:38781
I20260925 15:04:56.734798 30740 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:56.735183 30742 heartbeater.cc:347] Connected to a master server at 127.22.236.61:40973
I20260925 15:04:56.735329 30742 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:56.735611 30740 heartbeater.cc:511] Master 127.22.236.60:38781 requested a full tablet report, sending...
I20260925 15:04:56.735843 30742 heartbeater.cc:511] Master 127.22.236.61:40973 requested a full tablet report, sending...
I20260925 15:04:56.735206 30741 heartbeater.cc:347] Connected to a master server at 127.22.236.62:45627
I20260925 15:04:56.736637 30741 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:56.737044 30741 heartbeater.cc:511] Master 127.22.236.62:45627 requested a full tablet report, sending...
I20260925 15:04:56.737146 30416 ts_manager.cc:194] Registered new tserver with Master: 0b635f83f59d4b2b9f169b982f49c3f5 (127.22.236.2:38433)
I20260925 15:04:56.737196 30226 ts_manager.cc:194] Registered new tserver with Master: 0b635f83f59d4b2b9f169b982f49c3f5 (127.22.236.2:38433)
I20260925 15:04:56.737892 29622 ts_manager.cc:194] Re-registered known tserver with Master: 0b635f83f59d4b2b9f169b982f49c3f5 (127.22.236.2:38433)
I20260925 15:04:56.738662 29622 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:40485
I20260925 15:04:56.819573 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 29951
I20260925 15:04:56.832257 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:35359
--local_ip_for_outbound_sockets=127.22.236.3
--tserver_master_addrs=127.22.236.62:45627
--webserver_port=39719
--webserver_interface=127.22.236.3
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.3.keytab
--principal=oryx/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.236.62:45627,127.22.236.61:40973
--tserver_master_addrs=127.22.236.62:45627,127.22.236.61:40973,127.22.236.60:38781 with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:04:57.141685 30753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:04:57.141948 30753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:04:57.142073 30753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:04:57.149085 30753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:04:57.149204 30753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:04:57.149354 30753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:04:57.158970 30753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.3.keytab
--principal=oryx/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:35359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=39719
--tserver_master_addrs=127.22.236.62:45627,127.22.236.61:40973,127.22.236.60:38781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:04:57.160400 30753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:04:57.161625 30753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:04:57.175022 30759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:57.175442 30758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:04:57.175904 30761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:04:57.175935 30753 server_base.cc:1034] running on GCE node
I20260925 15:04:57.176597 30753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:04:57.177510 30753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:04:57.178710 30753 hybrid_clock.cc:648] HybridClock initialized: now 1790348697178651 us; error 48 us; skew 500 ppm
I20260925 15:04:57.179037 30753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:04:57 dist-test-slave-nsrc krb5kdc[29578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348697, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:04:57.182075 30753 init.cc:377] Logged in from keytab as oryx/127.22.236.3@KRBTEST.COM (short username oryx)
I20260925 15:04:57.183527 30753 webserver.cc:460] Webserver started at http://127.22.236.3:39719/ using document root <none> and password file <none>
I20260925 15:04:57.184221 30753 fs_manager.cc:359] Metadata directory not provided
I20260925 15:04:57.184343 30753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:04:57.189846 30753 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:04:57.192409 30768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:57.193728 30753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:04:57.193866 30753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "cd6d4fd224df40d48f64c3e8ee9446d7"
format_stamp: "Formatted at 2026-09-25 15:04:54 on dist-test-slave-nsrc"
server_key: "1480d366c37ef7145a1304b9260e0caa"
server_key_iv: "245ee94364c01d0d0c8f010ebe00c851"
server_key_version: "encryptionkey@0"
I20260925 15:04:57.194777 30753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:04:57.200862 30596 heartbeater.cc:503] Master 127.22.236.62:45627 was elected leader, sending a full tablet report...
I20260925 15:04:57.207759 30753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:04:57.211549 30753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:04:57.211789 30753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:04:57.212685 30753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:04:57.213898 30753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:04:57.213987 30753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:57.214084 30753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:04:57.214147 30753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:04:57.240855 30753 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:35359
I20260925 15:04:57.240916 30880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:35359 every 8 connection(s)
I20260925 15:04:57.242218 30753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:04:57.248867 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 30753
Sep 25 15:04:57 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348697, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.60@KRBTEST.COM
Sep 25 15:04:57 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348697, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.61@KRBTEST.COM
I20260925 15:04:57.275455 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.245941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:53507 (local address 127.22.236.61:40973)
0925 15:04:57.246432 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.246438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.255724 (+ 9286us) server_negotiation.cc:407] Connection header received
0925 15:04:57.255937 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.255943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.256021 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.256098 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:57.258991 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.260055 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.262209 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.262801 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.262949 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.267549 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:57.267556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:57.267562 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:57.267614 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:57.271004 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.272039 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.272045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.272049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.272239 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.272778 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.272784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.272788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.273145 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:57.273281 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.274815 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.275046 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":651,"threads_started":1}
I20260925 15:04:57.276213 30887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.246417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40973 (local address 127.22.236.3:53507)
0925 15:04:57.255626 (+ 9209us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:57.255643 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:57.255812 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:57.256327 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:57.256333 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:57.256960 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:57.257248 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.257260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.260249 (+ 2989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.260253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.262081 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.262092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.263158 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.263163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.263327 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.263911 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:57.263932 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:57.267311 (+ 3379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:57.271177 (+ 3866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:57.271188 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:57.271211 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:57.271880 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:57.272406 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:57.272411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:57.272415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:57.272650 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:57.273375 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:57.273384 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:57.273840 (+ 456us) client_negotiation.cc:769] Sending connection context
0925 15:04:57.274151 (+ 311us) client_negotiation.cc:240] Negotiation successful
0925 15:04:57.274782 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9095,"mutex_wait_us":290,"thread_start_us":338,"threads_started":1}
Sep 25 15:04:57 dist-test-slave-nsrc krb5kdc[29578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348697, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
I20260925 15:04:57.278163 30883 heartbeater.cc:347] Connected to a master server at 127.22.236.61:40973
I20260925 15:04:57.278468 30883 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:57.279341 30883 heartbeater.cc:511] Master 127.22.236.61:40973 requested a full tablet report, sending...
I20260925 15:04:57.281172 30226 ts_manager.cc:194] Registered new tserver with Master: cd6d4fd224df40d48f64c3e8ee9446d7 (127.22.236.3:35359)
I20260925 15:04:57.284276 30886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.245556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:38887 (local address 127.22.236.60:38781)
0925 15:04:57.246059 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.246066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.247353 (+ 1287us) server_negotiation.cc:407] Connection header received
0925 15:04:57.250281 (+ 2928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.250289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.250377 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.250457 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:57.253032 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.254269 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.257020 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.257673 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.257786 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.265839 (+ 8053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:57.265847 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:57.265853 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:57.265895 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:57.269990 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.278291 (+ 8301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.278295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.278299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.278438 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.279164 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.279169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.279172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.279754 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:57.279905 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.283675 (+ 3770us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.283908 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":350,"threads_started":1}
I20260925 15:04:57.285657 30890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.245509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:38781 (local address 127.22.236.3:38887)
0925 15:04:57.247179 (+ 1670us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:57.247230 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:57.249944 (+ 2714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:57.251002 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:57.251015 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:57.251893 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:57.252756 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.252780 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.254502 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.254508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.256840 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.256857 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.258054 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.258061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.258381 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.260360 (+ 1979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:57.260403 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:57.265569 (+ 5166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:57.274525 (+ 8956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:57.274531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:57.274535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:57.277972 (+ 3437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:57.278612 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:57.278616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:57.278619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:57.278774 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:57.283199 (+ 4425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:57.283204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:57.283378 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 15:04:57.284947 (+ 1569us) client_negotiation.cc:240] Negotiation successful
0925 15:04:57.285249 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1360,"thread_start_us":9768,"threads_started":1}
I20260925 15:04:57.286408 30881 heartbeater.cc:347] Connected to a master server at 127.22.236.60:38781
I20260925 15:04:57.286556 30881 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:57.286971 30881 heartbeater.cc:511] Master 127.22.236.60:38781 requested a full tablet report, sending...
I20260925 15:04:57.288069 30416 ts_manager.cc:194] Registered new tserver with Master: cd6d4fd224df40d48f64c3e8ee9446d7 (127.22.236.3:35359)
I20260925 15:04:57.290176 30888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.245773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.3:50667)
0925 15:04:57.254839 (+ 9066us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:57.254863 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:57.254982 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:57.258021 (+ 3039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:57.258025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:57.258567 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:57.258828 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.258837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.260216 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.260222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.262741 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.262755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.263664 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.263670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.264505 (+ 835us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.269349 (+ 4844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:57.269388 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:57.278441 (+ 9053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:57.282945 (+ 4504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:57.282950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:57.282954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:57.283471 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:57.284928 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:57.284934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:57.284938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:57.285106 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:57.289096 (+ 3990us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:57.289102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:57.289284 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 15:04:57.289448 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 15:04:57.289761 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8932,"mutex_wait_us":26,"thread_start_us":9272,"threads_started":1}
I20260925 15:04:57.290176 30885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.244923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:50667 (local address 127.22.236.62:45627)
0925 15:04:57.257701 (+ 12778us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.257708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.257726 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:04:57.257781 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.257786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.257867 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.257961 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:57.258973 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.260050 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.262932 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.263513 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.263626 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.278723 (+ 15097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:57.278730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:57.278736 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:57.278780 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:57.282838 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.284634 (+ 1796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.284641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.284644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.284828 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.288636 (+ 3808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.288641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.288645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.288973 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:57.289104 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.289600 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.289826 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12656,"thread_start_us":12548,"threads_started":1}
I20260925 15:04:57.290941 30882 heartbeater.cc:347] Connected to a master server at 127.22.236.62:45627
I20260925 15:04:57.291098 30882 heartbeater.cc:464] Registering TS with master...
I20260925 15:04:57.291446 30882 heartbeater.cc:511] Master 127.22.236.62:45627 requested a full tablet report, sending...
I20260925 15:04:57.292352 29622 ts_manager.cc:194] Re-registered known tserver with Master: cd6d4fd224df40d48f64c3e8ee9446d7 (127.22.236.3:35359)
I20260925 15:04:57.293470 29622 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:50667
I20260925 15:04:57.309691 30885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.265399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33810 (local address 127.22.236.62:45627)
0925 15:04:57.290565 (+ 25166us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.290571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.290587 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:04:57.290634 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.290639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.290712 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.290802 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:57.294204 (+ 3402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.295207 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.297307 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.297968 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.298096 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.302233 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:57.302240 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:57.302245 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:57.302282 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:57.305901 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.306769 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.306774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.306778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.306929 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.308231 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.308236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.308239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.308594 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:57.308832 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.309219 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.309388 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25064,"thread_start_us":35412,"threads_started":1}
I20260925 15:04:57.318269 30886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.264964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.22.236.60:38781)
0925 15:04:57.284551 (+ 19587us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.284556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.301520 (+ 16964us) server_negotiation.cc:407] Connection header received
0925 15:04:57.301783 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.301790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.301917 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.301981 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:57.305789 (+ 3808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.306809 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.308363 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.308933 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.309039 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.313300 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:57.313305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:57.313308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:57.313340 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:57.315897 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.316502 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.316508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.316519 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.316648 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.316975 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.316981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.316984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.317237 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:57.317444 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.317771 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.317968 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19490,"thread_start_us":40626,"threads_started":1}
I20260925 15:04:57.319526 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.277407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.22.236.61:40973)
0925 15:04:57.282279 (+ 4872us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.282285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.305462 (+ 23177us) server_negotiation.cc:407] Connection header received
0925 15:04:57.305562 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.305568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.305661 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.305751 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:57.307355 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.308115 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.309812 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.310231 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.310338 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.313535 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:57.313541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:57.313546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:57.313582 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:57.316523 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.317276 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.317282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.317285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.317439 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.317850 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.317855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.317859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.318222 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:57.318473 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.319006 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.319212 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4760}
I20260925 15:04:57.314942 29622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:04:57.323886 29622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:04:57.356305 30910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.345146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:46247 (local address 127.0.0.1:42988)
0925 15:04:57.346076 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:57.346097 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:57.346224 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:57.347866 (+ 1642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:57.347871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:57.347898 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:57.348323 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.348334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.350585 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.350591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.353366 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.353382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.355331 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.355337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.355515 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.355549 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:04:57.355635 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:04:57.355850 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":26,"thread_start_us":9607,"threads_started":1}
I20260925 15:04:57.356974 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.345503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42988 (local address 127.22.236.1:46247)
0925 15:04:57.347200 (+ 1697us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.347209 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.347239 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:04:57.347343 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.347351 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.347648 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.348528 (+ 880us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:57.348754 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.350386 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.353645 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.355169 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.355290 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.355492 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.356339 (+ 847us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.356691 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1546,"thread_start_us":1426,"threads_started":1}
I20260925 15:04:57.357508 30913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.345503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:35359 (local address 127.0.0.1:35306)
0925 15:04:57.347800 (+ 2297us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:57.347816 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:57.347937 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:57.349090 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:57.349096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:57.349121 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:57.349596 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.349611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.351679 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.351685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.354601 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.354615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.356818 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.356822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.356948 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.356969 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:04:57.357044 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:04:57.357159 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2178,"mutex_wait_us":25,"thread_start_us":691,"threads_started":1}
I20260925 15:04:57.357545 30912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.346049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.22.236.3:35359)
0925 15:04:57.346802 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.346812 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.348228 (+ 1416us) server_negotiation.cc:407] Connection header received
0925 15:04:57.348344 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.348353 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.348617 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.348805 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:57.349833 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.351479 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.354789 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.356418 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.356548 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.356720 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.357116 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.357260 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":395,"threads_started":1}
I20260925 15:04:57.360577 30816 tablet_service.cc:1467] Processing CreateTablet for tablet a30fb73f254142ac8482e3152797d2b7 (DEFAULT_TABLE table=test-table [id=bd6b99b5c2814500bc0bcf9c7198642e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:57.360683 30530 tablet_service.cc:1467] Processing CreateTablet for tablet a30fb73f254142ac8482e3152797d2b7 (DEFAULT_TABLE table=test-table [id=bd6b99b5c2814500bc0bcf9c7198642e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:57.361850 30816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a30fb73f254142ac8482e3152797d2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:57.361914 30530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a30fb73f254142ac8482e3152797d2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:57.362865 30909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.347331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:38433 (local address 127.0.0.1:58268)
0925 15:04:57.355017 (+ 7686us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:57.355034 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:57.355143 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:57.356047 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:57.356051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:57.356073 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:57.356586 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.356598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.358424 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.358429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.360286 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.360299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.362178 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.362184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.362344 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.362374 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:04:57.362446 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:04:57.362564 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7571,"thread_start_us":398,"threads_started":1}
I20260925 15:04:57.363543 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.347617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.22.236.2:38433)
0925 15:04:57.354649 (+ 7032us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.354657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.355567 (+ 910us) server_negotiation.cc:407] Connection header received
0925 15:04:57.355667 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.355675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.355876 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.356037 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:57.356747 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.358277 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.360451 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.361747 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.361876 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.362054 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.363152 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.363307 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6880,"thread_start_us":6788,"threads_started":1}
I20260925 15:04:57.367334 30675 tablet_service.cc:1467] Processing CreateTablet for tablet a30fb73f254142ac8482e3152797d2b7 (DEFAULT_TABLE table=test-table [id=bd6b99b5c2814500bc0bcf9c7198642e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:04:57.368626 30675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a30fb73f254142ac8482e3152797d2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:57.374490 30915 tablet_bootstrap.cc:492] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7: Bootstrap starting.
I20260925 15:04:57.376427 30916 tablet_bootstrap.cc:492] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a: Bootstrap starting.
I20260925 15:04:57.379110 30915 tablet_bootstrap.cc:654] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:57.379573 30916 tablet_bootstrap.cc:654] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:57.380503 30915 log.cc:824] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:57.381285 30916 log.cc:824] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:57.381489 30917 tablet_bootstrap.cc:492] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5: Bootstrap starting.
I20260925 15:04:57.384523 30917 tablet_bootstrap.cc:654] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:57.385389 30917 log.cc:824] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:57.387912 30917 tablet_bootstrap.cc:492] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5: No bootstrap required, opened a new log
I20260925 15:04:57.388186 30917 ts_tablet_manager.cc:1397] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260925 15:04:57.388319 30916 tablet_bootstrap.cc:492] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a: No bootstrap required, opened a new log
I20260925 15:04:57.388629 30916 ts_tablet_manager.cc:1397] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.010s
I20260925 15:04:57.390228 30915 tablet_bootstrap.cc:492] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7: No bootstrap required, opened a new log
I20260925 15:04:57.390522 30915 ts_tablet_manager.cc:1397] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260925 15:04:57.396791 30917 raft_consensus.cc:348] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:04:57.397159 30917 raft_consensus.cc:374] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:57.397248 30917 raft_consensus.cc:729] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b635f83f59d4b2b9f169b982f49c3f5, State: Initialized, Role: FOLLOWER
I20260925 15:04:57.397953 30917 consensus_queue.cc:260] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:04:57.397945 30916 raft_consensus.cc:348] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:04:57.398418 30916 raft_consensus.cc:374] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:57.398512 30916 raft_consensus.cc:729] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba42b62df1045df88a14643e5ac471a, State: Initialized, Role: FOLLOWER
I20260925 15:04:57.399044 30916 consensus_queue.cc:260] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:04:57.400987 30916 ts_tablet_manager.cc:1428] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260925 15:04:57.401051 30915 raft_consensus.cc:348] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:04:57.401412 30915 raft_consensus.cc:374] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:04:57.401486 30915 raft_consensus.cc:729] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6d4fd224df40d48f64c3e8ee9446d7, State: Initialized, Role: FOLLOWER
I20260925 15:04:57.402105 30915 consensus_queue.cc:260] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:04:57.408334 30915 ts_tablet_manager.cc:1428] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
I20260925 15:04:57.408351 30917 ts_tablet_manager.cc:1428] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5: Time spent starting tablet: real 0.020s user 0.010s sys 0.007s
I20260925 15:04:57.408598 30882 heartbeater.cc:503] Master 127.22.236.62:45627 was elected leader, sending a full tablet report...
I20260925 15:04:57.410722 30741 heartbeater.cc:503] Master 127.22.236.62:45627 was elected leader, sending a full tablet report...
W20260925 15:04:57.415737 30598 tablet.cc:2369] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:04:57.422492 30923 raft_consensus.cc:482] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:04:57.423123 30923 raft_consensus.cc:504] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:04:57.424732 30923 leader_election.cc:288] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ba42b62df1045df88a14643e5ac471a (127.22.236.1:46247), 0b635f83f59d4b2b9f169b982f49c3f5 (127.22.236.2:38433)
I20260925 15:04:57.429900 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.425257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:44975 (local address 127.22.236.2:38433)
0925 15:04:57.425436 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.425440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.425453 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:04:57.425508 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.425511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.425562 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.425631 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:57.425938 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.426775 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.428374 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.429110 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.429238 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.429307 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.429554 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.429668 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:04:57.429931 30890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.424945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:38433 (local address 127.22.236.3:44975)
0925 15:04:57.425189 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:57.425207 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:57.425299 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:57.425650 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:57.425654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:57.425671 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:57.425809 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.425815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.426887 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.426891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.428232 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.428247 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.429238 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.429244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.429414 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.429437 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:04:57.429523 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:04:57.429649 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":1}
I20260925 15:04:57.430022 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.424983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:54167 (local address 127.22.236.1:46247)
0925 15:04:57.425161 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.425166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.425187 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:04:57.425769 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.425772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.425823 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.425893 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:57.426245 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.426940 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.428456 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.429191 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.429320 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.429397 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.429675 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.429791 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:04:57.430228 30888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.424915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:46247 (local address 127.22.236.3:54167)
0925 15:04:57.425117 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:57.425141 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:57.425251 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:57.425936 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:57.425941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:57.425964 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:57.426104 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.426111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.427063 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.427067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.428325 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:57.428337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.429347 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.429352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:57.429543 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.429565 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:04:57.429638 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:04:57.429755 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 15:04:57.430842 30695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a30fb73f254142ac8482e3152797d2b7" candidate_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" is_pre_election: true
I20260925 15:04:57.431049 30550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a30fb73f254142ac8482e3152797d2b7" candidate_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba42b62df1045df88a14643e5ac471a" is_pre_election: true
I20260925 15:04:57.431376 30695 raft_consensus.cc:2454] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd6d4fd224df40d48f64c3e8ee9446d7 in term 0.
I20260925 15:04:57.431476 30550 raft_consensus.cc:2454] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd6d4fd224df40d48f64c3e8ee9446d7 in term 0.
I20260925 15:04:57.432018 30772 leader_election.cc:302] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b635f83f59d4b2b9f169b982f49c3f5, cd6d4fd224df40d48f64c3e8ee9446d7; no voters:
I20260925 15:04:57.432405 30923 raft_consensus.cc:2781] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:04:57.432534 30923 raft_consensus.cc:482] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:04:57.432623 30923 raft_consensus.cc:3037] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:57.435438 30923 raft_consensus.cc:504] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:04:57.435945 30923 leader_election.cc:288] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [CANDIDATE]: Term 1 election: Requested vote from peers 3ba42b62df1045df88a14643e5ac471a (127.22.236.1:46247), 0b635f83f59d4b2b9f169b982f49c3f5 (127.22.236.2:38433)
I20260925 15:04:57.436314 30550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a30fb73f254142ac8482e3152797d2b7" candidate_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba42b62df1045df88a14643e5ac471a"
I20260925 15:04:57.436321 30695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a30fb73f254142ac8482e3152797d2b7" candidate_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b635f83f59d4b2b9f169b982f49c3f5"
I20260925 15:04:57.436477 30550 raft_consensus.cc:3037] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:57.436479 30695 raft_consensus.cc:3037] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:57.438484 30550 raft_consensus.cc:2454] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd6d4fd224df40d48f64c3e8ee9446d7 in term 1.
I20260925 15:04:57.438480 30695 raft_consensus.cc:2454] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd6d4fd224df40d48f64c3e8ee9446d7 in term 1.
I20260925 15:04:57.438822 30772 leader_election.cc:302] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b635f83f59d4b2b9f169b982f49c3f5, 3ba42b62df1045df88a14643e5ac471a, cd6d4fd224df40d48f64c3e8ee9446d7; no voters:
I20260925 15:04:57.439112 30923 raft_consensus.cc:2781] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:04:57.439507 30923 raft_consensus.cc:686] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 1 LEADER]: Becoming Leader. State: Replica: cd6d4fd224df40d48f64c3e8ee9446d7, State: Running, Role: LEADER
I20260925 15:04:57.439993 30923 consensus_queue.cc:237] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:04:57.444591 29622 catalog_manager.cc:5495] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd6d4fd224df40d48f64c3e8ee9446d7 (127.22.236.3). New cstate: current_term: 1 leader_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } health_report { overall_health: UNKNOWN } } }
W20260925 15:04:57.460479 30743 tablet.cc:2369] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:04:57.464821 30912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.459522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.22.236.3:35359)
0925 15:04:57.459699 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.459704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.459804 (+ 100us) server_negotiation.cc:407] Connection header received
0925 15:04:57.460015 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.460018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.460091 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.460250 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:57.460569 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.461472 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.463083 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.463451 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.463536 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.463903 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:57.464409 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:57.464510 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.464568 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.464624 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:04:57.472345 30695 raft_consensus.cc:1261] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 1 FOLLOWER]: Refusing update from remote peer cd6d4fd224df40d48f64c3e8ee9446d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:57.472345 30550 raft_consensus.cc:1261] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a [term 1 FOLLOWER]: Refusing update from remote peer cd6d4fd224df40d48f64c3e8ee9446d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:57.473289 30923 consensus_queue.cc:1035] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:57.473558 30924 consensus_queue.cc:1035] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:57.488494 30928 mvcc.cc:205] Tried to move back new op lower bound from 7333268264837664768 to 7333268264717602816. Current Snapshot: MvccSnapshot[applied={T|T < 7333268264837664768}]
I20260925 15:04:57.504416 29623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:57.504634 29623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:04:57.511317 29623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5cdd435e9804a6c820218e27b67d29d: Sending DeleteTablet for 3 replicas of tablet a30fb73f254142ac8482e3152797d2b7
I20260925 15:04:57.512319 30530 tablet_service.cc:1514] Processing DeleteTablet for tablet a30fb73f254142ac8482e3152797d2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:57 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:42988
I20260925 15:04:57.512815 30675 tablet_service.cc:1514] Processing DeleteTablet for tablet a30fb73f254142ac8482e3152797d2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:57 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:58268
I20260925 15:04:57.512971 30816 tablet_service.cc:1514] Processing DeleteTablet for tablet a30fb73f254142ac8482e3152797d2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:04:57 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:35306
I20260925 15:04:57.515256 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 29591
I20260925 15:04:57.516186 29653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:57.530892 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.514066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.22.236.61:40973)
0925 15:04:57.514204 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.514210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.514227 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:04:57.514909 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.514914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.514985 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.515075 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:57.517675 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.518412 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.520185 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.520663 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.520771 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.524567 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:57.524574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:57.524579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:57.524611 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:57.527740 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.528578 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.528584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.528588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.528732 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.529159 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.529164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.529168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.529544 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:57.529756 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.530309 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.530536 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:04:57.540382 30937 tablet_replica.cc:330] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5: stopping tablet replica
I20260925 15:04:57.540882 30937 raft_consensus.cc:2229] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:57.541029 30936 tablet_replica.cc:330] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a: stopping tablet replica
I20260925 15:04:57.541548 30937 raft_consensus.cc:2258] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:57.541531 30936 raft_consensus.cc:2229] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:04:57.542016 30936 raft_consensus.cc:2258] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:57.542124 30938 tablet_replica.cc:330] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7: stopping tablet replica
I20260925 15:04:57.542548 30938 raft_consensus.cc:2229] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:04:57.543226 30938 raft_consensus.cc:2258] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:04:57.544380 30936 ts_tablet_manager.cc:1905] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:57.544690 30937 ts_tablet_manager.cc:1905] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:57.551035 30938 ts_tablet_manager.cc:1905] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:04:57.551342 30936 ts_tablet_manager.cc:1918] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:04:57.551561 30936 log.cc:1196] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/a30fb73f254142ac8482e3152797d2b7
I20260925 15:04:57.552189 30936 ts_tablet_manager.cc:1939] T a30fb73f254142ac8482e3152797d2b7 P 3ba42b62df1045df88a14643e5ac471a: Deleting consensus metadata
I20260925 15:04:57.557106 30886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.514640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59774 (local address 127.22.236.60:38781)
0925 15:04:57.514792 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.514799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.532226 (+ 17427us) server_negotiation.cc:407] Connection header received
0925 15:04:57.532476 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.532483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.532572 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.532644 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:57.534382 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.535465 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.537261 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.537963 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.538060 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.541620 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:57.541626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:57.541631 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:57.541669 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:57.544939 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.554601 (+ 9662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.554608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.554612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.554784 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.555284 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.555289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.555293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.555660 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:57.555899 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.556508 (+ 609us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.556757 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:04:57.560241 30938 ts_tablet_manager.cc:1918] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:04:57.560452 30937 ts_tablet_manager.cc:1918] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:04:57.560477 30938 log.cc:1196] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/a30fb73f254142ac8482e3152797d2b7
I20260925 15:04:57.560695 30937 log.cc:1196] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/a30fb73f254142ac8482e3152797d2b7
I20260925 15:04:57.561038 30938 ts_tablet_manager.cc:1939] T a30fb73f254142ac8482e3152797d2b7 P cd6d4fd224df40d48f64c3e8ee9446d7: Deleting consensus metadata
I20260925 15:04:57.561204 30937 ts_tablet_manager.cc:1939] T a30fb73f254142ac8482e3152797d2b7 P 0b635f83f59d4b2b9f169b982f49c3f5: Deleting consensus metadata
I20260925 15:04:57.663010 29607 catalog_manager.cc:4841] TS 3ba42b62df1045df88a14643e5ac471a (127.22.236.1:46247): tablet a30fb73f254142ac8482e3152797d2b7 (table test-table [id=bd6b99b5c2814500bc0bcf9c7198642e]) successfully deleted
I20260925 15:04:57.663091 29610 catalog_manager.cc:4841] TS 0b635f83f59d4b2b9f169b982f49c3f5 (127.22.236.2:38433): tablet a30fb73f254142ac8482e3152797d2b7 (table test-table [id=bd6b99b5c2814500bc0bcf9c7198642e]) successfully deleted
I20260925 15:04:57.663600 29609 catalog_manager.cc:4841] TS cd6d4fd224df40d48f64c3e8ee9446d7 (127.22.236.3:35359): tablet a30fb73f254142ac8482e3152797d2b7 (table test-table [id=bd6b99b5c2814500bc0bcf9c7198642e]) successfully deleted
I20260925 15:04:57.663734 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 29591
I20260925 15:04:57.727802 30886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.694285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59788 (local address 127.22.236.60:38781)
0925 15:04:57.694507 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.694527 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.706976 (+ 12449us) server_negotiation.cc:407] Connection header received
0925 15:04:57.707484 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.707490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.707568 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.707631 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:57.710060 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.711176 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.716307 (+ 5131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.716965 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.717066 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.720646 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:57.720653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:57.720658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:57.720697 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:57.724207 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.725220 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.725225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.725229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.725369 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.725808 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.725813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.725817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.726184 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:57.726412 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.727270 (+ 858us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.727476 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:04:57.730178 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:57.694310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.22.236.61:40973)
0925 15:04:57.694507 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:57.694511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:57.714199 (+ 19688us) server_negotiation.cc:407] Connection header received
0925 15:04:57.714421 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:57.714426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:57.714506 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:57.714577 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:57.716762 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.717882 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.720028 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:57.720787 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:57.720904 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:57.723896 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:57.723903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:57.723908 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:57.723946 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:57.727065 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.727766 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.727771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.727774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.727907 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:57.728373 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:57.728379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:57.728383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:57.728757 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:57.728986 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:57.729595 (+ 609us) server_negotiation.cc:299] Negotiation successful
0925 15:04:57.729827 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:04:58.492926 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:58.492215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.2:52829)
0925 15:04:58.492608 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:58.492758 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":190,"threads_started":1}
W20260925 15:04:58.493345 30741 heartbeater.cc:644] Failed to heartbeat to 127.22.236.62:45627 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
I20260925 15:04:58.495587 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:58.494836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.3:34501)
0925 15:04:58.495252 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:58.495412 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":226,"threads_started":1}
W20260925 15:04:58.495929 30882 heartbeater.cc:644] Failed to heartbeat to 127.22.236.62:45627 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
I20260925 15:04:58.498867 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:58.497658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.1:35369)
0925 15:04:58.498647 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:58.498738 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":785,"threads_started":1}
W20260925 15:04:58.499238 30596 heartbeater.cc:644] Failed to heartbeat to 127.22.236.62:45627 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
I20260925 15:04:59.011150 30951 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d5cdd435e9804a6c820218e27b67d29d)
I20260925 15:04:59.011390 30952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d5cdd435e9804a6c820218e27b67d29d)
I20260925 15:04:59.011397 30951 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } }
I20260925 15:04:59.011618 30952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } }
I20260925 15:04:59.018855 30951 leader_election.cc:288] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5cdd435e9804a6c820218e27b67d29d (127.22.236.62:45627), 69619aa463594ba086f18ec78c678cc4 (127.22.236.61:40973)
I20260925 15:04:59.019011 30952 leader_election.cc:288] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5cdd435e9804a6c820218e27b67d29d (127.22.236.62:45627), bf3d83eb24d040468c204d3217258138 (127.22.236.60:38781)
I20260925 15:04:59.024269 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.012956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34208)
0925 15:04:59.023712 (+ 10756us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:59.023964 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10614,"mutex_wait_us":3,"threads_started":1}
W20260925 15:04:59.024863 30210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
I20260925 15:04:59.027164 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.026752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34220)
0925 15:04:59.026933 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:59.027024 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260925 15:04:59.027484 30210 leader_election.cc:334] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d5cdd435e9804a6c820218e27b67d29d (127.22.236.62:45627): Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
I20260925 15:04:59.027813 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.012946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34206)
0925 15:04:59.027362 (+ 14416us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:59.027568 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13459,"thread_start_us":13922,"threads_started":1}
W20260925 15:04:59.028304 30400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
I20260925 15:04:59.030030 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.029721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34234)
0925 15:04:59.029850 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:59.029931 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:04:59.030221 30400 leader_election.cc:334] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d5cdd435e9804a6c820218e27b67d29d (127.22.236.62:45627): Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
I20260925 15:04:59.033654 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.012955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40973 (local address 127.0.0.1:44100)
0925 15:04:59.027102 (+ 14147us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:59.027120 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:59.027262 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:59.028422 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:59.028427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:59.028475 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:59.028828 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:59.028844 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:59.030426 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:59.030432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:59.031741 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:59.031753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:59.032858 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:59.032864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:59.033036 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:59.033058 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:04:59.033132 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:04:59.033253 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14037,"mutex_wait_us":5,"spinlock_wait_cycles":2816,"thread_start_us":13613,"threads_started":1}
I20260925 15:04:59.033685 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.023701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.22.236.61:40973)
0925 15:04:59.028134 (+ 4433us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:59.028159 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:59.028178 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:04:59.028240 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:59.028244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:59.028297 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:59.028381 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:59.029059 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:59.030256 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:59.031881 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:59.032712 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:59.032818 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:59.032928 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:59.033168 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 15:04:59.033324 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":4348,"thread_start_us":4144,"threads_started":1}
I20260925 15:04:59.033725 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.012946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:38781 (local address 127.0.0.1:59800)
0925 15:04:59.023647 (+ 10701us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:59.023668 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:59.023812 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:59.028645 (+ 4833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:59.028650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:59.028671 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:04:59.028958 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:59.028969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:59.030534 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:59.030539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:59.031768 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:59.031794 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:59.032880 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:59.032886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:59.033066 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:59.033089 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:04:59.033164 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:04:59.033312 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10581,"mutex_wait_us":3,"thread_start_us":10430,"threads_started":1}
I20260925 15:04:59.033682 30959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.027178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.22.236.60:38781)
0925 15:04:59.028309 (+ 1131us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:59.028315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:59.028334 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:04:59.028400 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:59.028404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:59.028501 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:59.028621 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:04:59.029189 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:59.030426 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:59.031924 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:59.032727 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:59.032836 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:59.032982 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:59.033195 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 15:04:59.033312 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":1018,"spinlock_wait_cycles":34432,"thread_start_us":882,"threads_started":1}
I20260925 15:04:59.034384 30426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69619aa463594ba086f18ec78c678cc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bf3d83eb24d040468c204d3217258138" is_pre_election: true
I20260925 15:04:59.034469 30236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf3d83eb24d040468c204d3217258138" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "69619aa463594ba086f18ec78c678cc4" is_pre_election: true
I20260925 15:04:59.034660 30426 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69619aa463594ba086f18ec78c678cc4 in term 1.
I20260925 15:04:59.034723 30236 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf3d83eb24d040468c204d3217258138 in term 1.
I20260925 15:04:59.035125 30403 leader_election.cc:302] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69619aa463594ba086f18ec78c678cc4, bf3d83eb24d040468c204d3217258138; no voters: d5cdd435e9804a6c820218e27b67d29d
I20260925 15:04:59.035127 30211 leader_election.cc:302] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69619aa463594ba086f18ec78c678cc4, bf3d83eb24d040468c204d3217258138; no voters: d5cdd435e9804a6c820218e27b67d29d
I20260925 15:04:59.035439 30951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 15:04:59.035571 30951 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d5cdd435e9804a6c820218e27b67d29d)
I20260925 15:04:59.035660 30951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:04:59.035641 30952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 15:04:59.035745 30952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d5cdd435e9804a6c820218e27b67d29d)
I20260925 15:04:59.035831 30952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:04:59.037707 30951 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } }
I20260925 15:04:59.037781 30952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } }
I20260925 15:04:59.038257 30951 leader_election.cc:288] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [CANDIDATE]: Term 2 election: Requested vote from peers d5cdd435e9804a6c820218e27b67d29d (127.22.236.62:45627), 69619aa463594ba086f18ec78c678cc4 (127.22.236.61:40973)
I20260925 15:04:59.038369 30952 leader_election.cc:288] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [CANDIDATE]: Term 2 election: Requested vote from peers d5cdd435e9804a6c820218e27b67d29d (127.22.236.62:45627), bf3d83eb24d040468c204d3217258138 (127.22.236.60:38781)
I20260925 15:04:59.038764 30236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf3d83eb24d040468c204d3217258138" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "69619aa463594ba086f18ec78c678cc4"
I20260925 15:04:59.038995 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.038650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34240)
0925 15:04:59.038779 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:59.038857 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:04:59.039043 30236 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bf3d83eb24d040468c204d3217258138 in current term 2: Already voted for candidate 69619aa463594ba086f18ec78c678cc4 in this term.
I20260925 15:04:59.038996 30426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69619aa463594ba086f18ec78c678cc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bf3d83eb24d040468c204d3217258138"
I20260925 15:04:59.039265 30426 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 69619aa463594ba086f18ec78c678cc4 in current term 2: Already voted for candidate bf3d83eb24d040468c204d3217258138 in this term.
I20260925 15:04:59.039431 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.038446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34248)
0925 15:04:59.038602 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:59.039303 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 15:04:59.040390 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.040015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34260)
0925 15:04:59.040161 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:59.040249 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:04:59.040402 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.040060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34276)
0925 15:04:59.040213 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:59.040275 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:04:59.040560 30210 leader_election.cc:334] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d5cdd435e9804a6c820218e27b67d29d (127.22.236.62:45627): Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
W20260925 15:04:59.040585 30400 leader_election.cc:334] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d5cdd435e9804a6c820218e27b67d29d (127.22.236.62:45627): Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
I20260925 15:04:59.040649 30210 leader_election.cc:302] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69619aa463594ba086f18ec78c678cc4; no voters: bf3d83eb24d040468c204d3217258138, d5cdd435e9804a6c820218e27b67d29d
I20260925 15:04:59.040679 30400 leader_election.cc:302] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf3d83eb24d040468c204d3217258138; no voters: 69619aa463594ba086f18ec78c678cc4, d5cdd435e9804a6c820218e27b67d29d
I20260925 15:04:59.040865 30952 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 15:04:59.040977 30951 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 15:04:59.494611 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.493988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.2:34017)
0925 15:04:59.494378 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:59.494456 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":200,"threads_started":1}
I20260925 15:04:59.497290 30962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.496523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.3:37717)
0925 15:04:59.496967 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:59.497058 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":221,"threads_started":1}
I20260925 15:04:59.500422 30963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:59.499795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.1:59879)
0925 15:04:59.500172 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:59.500254 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":180,"threads_started":1}
I20260925 15:05:00.496294 30964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:00.495530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.2:57297)
0925 15:05:00.495938 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:00.496034 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":194,"threads_started":1}
I20260925 15:05:00.498857 30965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:00.498012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.3:56789)
0925 15:05:00.498593 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:00.498688 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":367,"threads_started":1}
I20260925 15:05:00.501720 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:00.501134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.1:41281)
0925 15:05:00.501512 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:00.501586 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":187,"threads_started":1}
I20260925 15:05:01.112174 30967 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:05:01.112383 30967 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } }
I20260925 15:05:01.113023 30967 leader_election.cc:288] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d5cdd435e9804a6c820218e27b67d29d (127.22.236.62:45627), 69619aa463594ba086f18ec78c678cc4 (127.22.236.61:40973)
I20260925 15:05:01.113564 30236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf3d83eb24d040468c204d3217258138" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "69619aa463594ba086f18ec78c678cc4" is_pre_election: true
I20260925 15:05:01.113773 30236 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf3d83eb24d040468c204d3217258138 in term 2.
I20260925 15:05:01.113977 30968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.113229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34584)
0925 15:05:01.113677 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.113786 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":242,"threads_started":1}
I20260925 15:05:01.114199 30403 leader_election.cc:302] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69619aa463594ba086f18ec78c678cc4, bf3d83eb24d040468c204d3217258138; no voters:
I20260925 15:05:01.114444 30967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 15:05:01.114564 30967 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:05:01.114663 30967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:05:01.115468 30968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.115070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34598)
0925 15:05:01.115244 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.115337 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:05:01.115644 30400 leader_election.cc:334] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d5cdd435e9804a6c820218e27b67d29d (127.22.236.62:45627): Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
I20260925 15:05:01.116951 30967 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } }
I20260925 15:05:01.117427 30967 leader_election.cc:288] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [CANDIDATE]: Term 3 election: Requested vote from peers d5cdd435e9804a6c820218e27b67d29d (127.22.236.62:45627), 69619aa463594ba086f18ec78c678cc4 (127.22.236.61:40973)
I20260925 15:05:01.117893 30968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.117554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34604)
0925 15:05:01.117677 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.117742 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 15:05:01.117882 30236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf3d83eb24d040468c204d3217258138" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "69619aa463594ba086f18ec78c678cc4"
I20260925 15:05:01.118048 30236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:05:01.118723 30968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.118472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34608)
0925 15:05:01.118577 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.118636 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 15:05:01.118852 30400 leader_election.cc:334] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d5cdd435e9804a6c820218e27b67d29d (127.22.236.62:45627): Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
I20260925 15:05:01.120095 30236 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf3d83eb24d040468c204d3217258138 in term 3.
I20260925 15:05:01.120496 30403 leader_election.cc:302] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69619aa463594ba086f18ec78c678cc4, bf3d83eb24d040468c204d3217258138; no voters: d5cdd435e9804a6c820218e27b67d29d
I20260925 15:05:01.120805 30967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:05:01.121215 30967 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [term 3 LEADER]: Becoming Leader. State: Replica: bf3d83eb24d040468c204d3217258138, State: Running, Role: LEADER
I20260925 15:05:01.121642 30967 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } }
I20260925 15:05:01.123245 30970 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf3d83eb24d040468c204d3217258138. Latest consensus state: current_term: 3 leader_uuid: "bf3d83eb24d040468c204d3217258138" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } } }
I20260925 15:05:01.123490 30970 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [sys.catalog]: This master's current role is: LEADER
I20260925 15:05:01.124024 30972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:05:01.127327 30972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:05:01.128072 30972 catalog_manager.cc:1259] Loaded cluster ID: d8f4c53ce2424dabaec159f4fffbc069
I20260925 15:05:01.128159 30972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:05:01.129242 30972 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:05:01.129992 30972 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: Loaded TSK: 0
I20260925 15:05:01.130769 30972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:05:01.145080 30415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:05:01.151844 30415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:05:01.155963 30968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.155565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34628)
0925 15:05:01.155745 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.155817 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 15:05:01.155977 30236 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [term 3 FOLLOWER]: Refusing update from remote peer bf3d83eb24d040468c204d3217258138: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 15:05:01.156621 30970 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 15:05:01.156963 30968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.156605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34644)
0925 15:05:01.156773 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.156842 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 15:05:01.157125 30400 consensus_peers.cc:487] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 -> Peer d5cdd435e9804a6c820218e27b67d29d (127.22.236.62:45627): Couldn't send request to peer d5cdd435e9804a6c820218e27b67d29d. Status: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 15:05:01.158995 30974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf3d83eb24d040468c204d3217258138. Latest consensus state: current_term: 3 leader_uuid: "bf3d83eb24d040468c204d3217258138" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } } }
I20260925 15:05:01.159219 30974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:05:01.160732 30967 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bf3d83eb24d040468c204d3217258138" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } } }
I20260925 15:05:01.161012 30967 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138 [sys.catalog]: This master's current role is: LEADER
I20260925 15:05:01.162325 30975 mvcc.cc:205] Tried to move back new op lower bound from 7333268279928848384 to 7333268279797243904. Current Snapshot: MvccSnapshot[applied={T|T < 7333268279928848384}]
I20260925 15:05:01.165470 30974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bf3d83eb24d040468c204d3217258138" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5cdd435e9804a6c820218e27b67d29d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45627 } } peers { permanent_uuid: "69619aa463594ba086f18ec78c678cc4" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40973 } attrs { promote: false } } peers { permanent_uuid: "bf3d83eb24d040468c204d3217258138" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 38781 } attrs { promote: false } } }
I20260925 15:05:01.165710 30974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69619aa463594ba086f18ec78c678cc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:05:01.183467 30968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.170410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:35359 (local address 127.0.0.1:35326)
0925 15:05:01.170569 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.170589 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:01.170708 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:01.176856 (+ 6148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:01.176862 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:01.176906 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:01.177283 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:01.177295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.178844 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.178850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:01.180983 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:01.180999 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.182663 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.182670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:01.182888 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:01.182928 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 15:05:01.182997 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:05:01.183143 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
I20260925 15:05:01.184494 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.172294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.22.236.3:35359)
0925 15:05:01.176515 (+ 4221us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:01.176522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:01.176542 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:05:01.176616 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:01.176621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:01.176696 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:01.177421 (+ 725us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:01.177581 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.178676 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.181194 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.182450 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.183814 (+ 1364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:01.183948 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:01.184005 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 15:05:01.184170 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4081,"thread_start_us":420,"threads_started":1}
I20260925 15:05:01.185312 30983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.172617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:38433 (local address 127.0.0.1:58282)
0925 15:05:01.173062 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.173078 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:01.173182 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:01.176766 (+ 3584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:01.176771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:01.176793 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:01.177184 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:01.177196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.179800 (+ 2604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.179808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:01.181933 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:01.181946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.184540 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.184546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:01.184726 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:01.184759 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:05:01.184830 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:05:01.184959 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"spinlock_wait_cycles":768,"thread_start_us":3600,"threads_started":1}
I20260925 15:05:01.185781 30816 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae82b1dcd3d49c7924f9ba7a7557755 (DEFAULT_TABLE table=test-table [id=06f9e43585d04f33a64ba728a8662156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:01.186223 30984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.172764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.22.236.2:38433)
0925 15:05:01.176460 (+ 3696us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:01.176466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:01.176484 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:05:01.176561 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:01.176567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:01.176638 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:01.177387 (+ 749us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:01.177536 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.179643 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.182145 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.184319 (+ 2174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.185703 (+ 1384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:01.185805 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:01.185852 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 15:05:01.185985 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3586,"thread_start_us":3495,"threads_started":1}
I20260925 15:05:01.186293 30816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae82b1dcd3d49c7924f9ba7a7557755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:01.186775 30675 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae82b1dcd3d49c7924f9ba7a7557755 (DEFAULT_TABLE table=test-table [id=06f9e43585d04f33a64ba728a8662156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:01.187163 30675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae82b1dcd3d49c7924f9ba7a7557755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:01.195235 30981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.171145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.22.236.1:46247)
0925 15:05:01.187547 (+ 16402us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:01.187554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:01.187574 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:05:01.187639 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:01.187644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:01.187729 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:01.187825 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:01.188833 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.189931 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.192250 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.193595 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.193732 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:01.193856 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:01.194902 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0925 15:05:01.195045 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16287,"thread_start_us":16202,"threads_started":1}
I20260925 15:05:01.195264 30980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.170830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:46247 (local address 127.0.0.1:42994)
0925 15:05:01.171329 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.171344 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:01.171440 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:01.188287 (+ 16847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:01.188292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:01.188319 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:01.188658 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:01.188670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.190107 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.190113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:01.192067 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:01.192083 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.194571 (+ 2488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.194578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:01.194745 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:01.194776 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:05:01.194865 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:05:01.194994 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":425,"threads_started":1}
I20260925 15:05:01.195511 30985 tablet_bootstrap.cc:492] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7: Bootstrap starting.
I20260925 15:05:01.196098 30530 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae82b1dcd3d49c7924f9ba7a7557755 (DEFAULT_TABLE table=test-table [id=06f9e43585d04f33a64ba728a8662156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:01.196496 30530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae82b1dcd3d49c7924f9ba7a7557755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:01.196920 30985 tablet_bootstrap.cc:654] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:01.197707 30986 tablet_bootstrap.cc:492] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5: Bootstrap starting.
I20260925 15:05:01.199074 30986 tablet_bootstrap.cc:654] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:01.200687 30985 tablet_bootstrap.cc:492] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7: No bootstrap required, opened a new log
I20260925 15:05:01.200809 30985 ts_tablet_manager.cc:1397] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 15:05:01.201493 30985 raft_consensus.cc:348] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:05:01.202219 30986 tablet_bootstrap.cc:492] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5: No bootstrap required, opened a new log
I20260925 15:05:01.202340 30986 ts_tablet_manager.cc:1397] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260925 15:05:01.202884 30986 raft_consensus.cc:348] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:05:01.203063 30986 raft_consensus.cc:374] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:01.203147 30986 raft_consensus.cc:729] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b635f83f59d4b2b9f169b982f49c3f5, State: Initialized, Role: FOLLOWER
I20260925 15:05:01.203365 30986 consensus_queue.cc:260] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:05:01.203678 30985 raft_consensus.cc:374] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:01.203785 30985 raft_consensus.cc:729] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6d4fd224df40d48f64c3e8ee9446d7, State: Initialized, Role: FOLLOWER
I20260925 15:05:01.203989 30985 consensus_queue.cc:260] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:05:01.204610 30986 ts_tablet_manager.cc:1428] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:05:01.207072 30740 heartbeater.cc:503] Master 127.22.236.60:38781 was elected leader, sending a full tablet report...
I20260925 15:05:01.208037 30985 ts_tablet_manager.cc:1428] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260925 15:05:01.210194 30881 heartbeater.cc:503] Master 127.22.236.60:38781 was elected leader, sending a full tablet report...
I20260925 15:05:01.213569 30989 tablet_bootstrap.cc:492] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a: Bootstrap starting.
I20260925 15:05:01.215703 30989 tablet_bootstrap.cc:654] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:01.217685 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.204311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.2:55663)
0925 15:05:01.217238 (+ 12927us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.217422 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12777,"threads_started":1}
I20260925 15:05:01.218153 30989 tablet_bootstrap.cc:492] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a: No bootstrap required, opened a new log
I20260925 15:05:01.218278 30989 ts_tablet_manager.cc:1397] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260925 15:05:01.218909 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.218524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.2:37503)
0925 15:05:01.218682 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.218778 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 15:05:01.218956 30989 raft_consensus.cc:348] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:05:01.219168 30989 raft_consensus.cc:374] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:01.219254 30989 raft_consensus.cc:729] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba42b62df1045df88a14643e5ac471a, State: Initialized, Role: FOLLOWER
I20260925 15:05:01.219486 30989 consensus_queue.cc:260] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:05:01.220242 30989 ts_tablet_manager.cc:1428] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:05:01.222033 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.209502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.3:41313)
0925 15:05:01.221634 (+ 12132us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.221778 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11995,"threads_started":1}
I20260925 15:05:01.223121 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.222689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.3:45293)
0925 15:05:01.222878 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.222988 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 15:05:01.223412 30595 heartbeater.cc:503] Master 127.22.236.60:38781 was elected leader, sending a full tablet report...
I20260925 15:05:01.223745 30996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.220744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.1:35207)
0925 15:05:01.223204 (+ 2460us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.223463 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2325,"thread_start_us":2208,"threads_started":1}
I20260925 15:05:01.224561 30996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.224271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.1:57633)
0925 15:05:01.224391 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.224454 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 15:05:01.254464 30884 tablet.cc:2369] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:05:01.493785 30991 raft_consensus.cc:482] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:05:01.494081 30991 raft_consensus.cc:504] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:05:01.494769 30991 leader_election.cc:288] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ba42b62df1045df88a14643e5ac471a (127.22.236.1:46247), 0b635f83f59d4b2b9f169b982f49c3f5 (127.22.236.2:38433)
I20260925 15:05:01.495409 30550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae82b1dcd3d49c7924f9ba7a7557755" candidate_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba42b62df1045df88a14643e5ac471a" is_pre_election: true
I20260925 15:05:01.495397 30695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae82b1dcd3d49c7924f9ba7a7557755" candidate_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" is_pre_election: true
I20260925 15:05:01.495653 30695 raft_consensus.cc:2454] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd6d4fd224df40d48f64c3e8ee9446d7 in term 0.
I20260925 15:05:01.495653 30550 raft_consensus.cc:2454] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd6d4fd224df40d48f64c3e8ee9446d7 in term 0.
I20260925 15:05:01.496169 30769 leader_election.cc:302] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b635f83f59d4b2b9f169b982f49c3f5, 3ba42b62df1045df88a14643e5ac471a, cd6d4fd224df40d48f64c3e8ee9446d7; no voters:
I20260925 15:05:01.496464 30991 raft_consensus.cc:2781] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:05:01.496582 30991 raft_consensus.cc:482] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:05:01.496661 30991 raft_consensus.cc:3037] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:01.498813 30991 raft_consensus.cc:504] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:05:01.499279 30991 leader_election.cc:288] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [CANDIDATE]: Term 1 election: Requested vote from peers 3ba42b62df1045df88a14643e5ac471a (127.22.236.1:46247), 0b635f83f59d4b2b9f169b982f49c3f5 (127.22.236.2:38433)
I20260925 15:05:01.499806 30695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae82b1dcd3d49c7924f9ba7a7557755" candidate_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b635f83f59d4b2b9f169b982f49c3f5"
I20260925 15:05:01.499792 30550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae82b1dcd3d49c7924f9ba7a7557755" candidate_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba42b62df1045df88a14643e5ac471a"
I20260925 15:05:01.500000 30695 raft_consensus.cc:3037] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:01.500000 30550 raft_consensus.cc:3037] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:01.502794 30550 raft_consensus.cc:2454] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd6d4fd224df40d48f64c3e8ee9446d7 in term 1.
I20260925 15:05:01.502794 30695 raft_consensus.cc:2454] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd6d4fd224df40d48f64c3e8ee9446d7 in term 1.
I20260925 15:05:01.503187 30772 leader_election.cc:302] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b635f83f59d4b2b9f169b982f49c3f5, cd6d4fd224df40d48f64c3e8ee9446d7; no voters:
I20260925 15:05:01.503458 30991 raft_consensus.cc:2781] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:05:01.503803 30991 raft_consensus.cc:686] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 1 LEADER]: Becoming Leader. State: Replica: cd6d4fd224df40d48f64c3e8ee9446d7, State: Running, Role: LEADER
I20260925 15:05:01.504065 30991 consensus_queue.cc:237] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } } peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } }
I20260925 15:05:01.504920 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.504454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.3:52165)
0925 15:05:01.504611 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.504699 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:05:01.506575 30416 catalog_manager.cc:5495] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd6d4fd224df40d48f64c3e8ee9446d7 (127.22.236.3). New cstate: current_term: 1 leader_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6d4fd224df40d48f64c3e8ee9446d7" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 } health_report { overall_health: UNKNOWN } } }
I20260925 15:05:01.532392 30980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.531939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34650)
0925 15:05:01.532097 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.532243 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 15:05:01.533334 30980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.533004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34666)
0925 15:05:01.533142 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.533218 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:05:01.544821 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.540494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.22.236.3:35359)
0925 15:05:01.540703 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:01.540709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:01.540729 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:05:01.540795 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:01.540801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:01.540872 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:01.540943 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:05:01.541294 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.542039 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.543321 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.543696 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.543798 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:01.544187 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:05:01.544287 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:05:01.544360 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:01.544498 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 15:05:01.544569 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:05:01.547530 30695 raft_consensus.cc:1261] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 1 FOLLOWER]: Refusing update from remote peer cd6d4fd224df40d48f64c3e8ee9446d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:05:01.547708 30550 raft_consensus.cc:1261] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a [term 1 FOLLOWER]: Refusing update from remote peer cd6d4fd224df40d48f64c3e8ee9446d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:05:01.548168 30997 consensus_queue.cc:1035] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b635f83f59d4b2b9f169b982f49c3f5" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:05:01.548498 30991 consensus_queue.cc:1035] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba42b62df1045df88a14643e5ac471a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 46247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:05:01.550441 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.549909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.2:37809)
0925 15:05:01.550130 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.550218 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 15:05:01.551548 30996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.551134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.1:34431)
0925 15:05:01.551277 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.551384 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:05:01.554325 31000 mvcc.cc:205] Tried to move back new op lower bound from 7333268281533591552 to 7333268281362980864. Current Snapshot: MvccSnapshot[applied={T|T < 7333268281533591552}]
I20260925 15:05:01.556782 30996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.556291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.1:55865)
0925 15:05:01.556563 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.556643 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260925 15:05:01.557929 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.556537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.3:60019)
0925 15:05:01.557673 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.557776 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1016}
I20260925 15:05:01.559890 31002 mvcc.cc:205] Tried to move back new op lower bound from 7333268281533591552 to 7333268281362980864. Current Snapshot: MvccSnapshot[applied={T|T < 7333268281533591552}]
I20260925 15:05:01.565162 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.564621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.2:35221)
0925 15:05:01.564928 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.565013 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260925 15:05:01.565742 30416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:05:01.565979 30416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:05:01.568261 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.567857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.22.236.3:44833)
0925 15:05:01.567987 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.568069 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 15:05:01.572634 30416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf3d83eb24d040468c204d3217258138: Sending DeleteTablet for 3 replicas of tablet 5ae82b1dcd3d49c7924f9ba7a7557755
I20260925 15:05:01.573341 30675 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae82b1dcd3d49c7924f9ba7a7557755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:01 UTC) from {username='oryx', principal='oryx/127.22.236.60@KRBTEST.COM'} at 127.0.0.1:58282
I20260925 15:05:01.573405 30816 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae82b1dcd3d49c7924f9ba7a7557755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:01 UTC) from {username='oryx', principal='oryx/127.22.236.60@KRBTEST.COM'} at 127.0.0.1:35326
I20260925 15:05:01.573972 31009 tablet_replica.cc:330] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5: stopping tablet replica
I20260925 15:05:01.574163 31010 tablet_replica.cc:330] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7: stopping tablet replica
I20260925 15:05:01.574177 31009 raft_consensus.cc:2229] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:05:01.574465 31009 raft_consensus.cc:2258] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:01.574571 30530 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae82b1dcd3d49c7924f9ba7a7557755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:01 UTC) from {username='oryx', principal='oryx/127.22.236.60@KRBTEST.COM'} at 127.0.0.1:42994
I20260925 15:05:01.574846 30675 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae82b1dcd3d49c7924f9ba7a7557755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:01 UTC) from {username='oryx', principal='oryx/127.22.236.60@KRBTEST.COM'} at 127.0.0.1:58282
I20260925 15:05:01.575991 31009 ts_tablet_manager.cc:1905] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:01.576889 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 30388
I20260925 15:05:01.581166 31010 raft_consensus.cc:2229] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:05:01.581542 31010 raft_consensus.cc:2258] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:01.581769 31011 tablet_replica.cc:330] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a: stopping tablet replica
I20260925 15:05:01.581981 31011 raft_consensus.cc:2229] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:05:01.582284 31011 raft_consensus.cc:2258] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:01.582710 31010 ts_tablet_manager.cc:1905] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:01.583425 31009 ts_tablet_manager.cc:1918] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:05:01.583590 31009 log.cc:1196] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/5ae82b1dcd3d49c7924f9ba7a7557755
I20260925 15:05:01.583659 31011 ts_tablet_manager.cc:1905] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:01.584036 31009 ts_tablet_manager.cc:1939] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 0b635f83f59d4b2b9f169b982f49c3f5: Deleting consensus metadata
I20260925 15:05:01.587432 30403 catalog_manager.cc:4841] TS 0b635f83f59d4b2b9f169b982f49c3f5 (127.22.236.2:38433): tablet 5ae82b1dcd3d49c7924f9ba7a7557755 (table test-table [id=06f9e43585d04f33a64ba728a8662156]) successfully deleted
W20260925 15:05:01.587908 30403 catalog_manager.cc:4808] TS 0b635f83f59d4b2b9f169b982f49c3f5 (127.22.236.2:38433): delete failed for tablet 5ae82b1dcd3d49c7924f9ba7a7557755 because the tablet was not found. No further retry: Not found: Tablet not found: 5ae82b1dcd3d49c7924f9ba7a7557755
I20260925 15:05:01.588335 31010 ts_tablet_manager.cc:1918] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:05:01.588479 31010 log.cc:1196] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/5ae82b1dcd3d49c7924f9ba7a7557755
I20260925 15:05:01.588917 31010 ts_tablet_manager.cc:1939] T 5ae82b1dcd3d49c7924f9ba7a7557755 P cd6d4fd224df40d48f64c3e8ee9446d7: Deleting consensus metadata
I20260925 15:05:01.589869 30402 catalog_manager.cc:4841] TS cd6d4fd224df40d48f64c3e8ee9446d7 (127.22.236.3:35359): tablet 5ae82b1dcd3d49c7924f9ba7a7557755 (table test-table [id=06f9e43585d04f33a64ba728a8662156]) successfully deleted
I20260925 15:05:01.593197 31011 ts_tablet_manager.cc:1918] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:05:01.593315 31011 log.cc:1196] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/5ae82b1dcd3d49c7924f9ba7a7557755
I20260925 15:05:01.593698 31011 ts_tablet_manager.cc:1939] T 5ae82b1dcd3d49c7924f9ba7a7557755 P 3ba42b62df1045df88a14643e5ac471a: Deleting consensus metadata
I20260925 15:05:01.594523 30400 catalog_manager.cc:4841] TS 3ba42b62df1045df88a14643e5ac471a (127.22.236.1:46247): tablet 5ae82b1dcd3d49c7924f9ba7a7557755 (table test-table [id=06f9e43585d04f33a64ba728a8662156]) successfully deleted
I20260925 15:05:01.597239 30981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.578476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.22.236.1:46247)
0925 15:05:01.578668 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:01.578674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:01.582821 (+ 4147us) server_negotiation.cc:407] Connection header received
0925 15:05:01.583099 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:01.583108 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:01.583192 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:01.583262 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:01.585108 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.586069 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.587694 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.588369 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.588496 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:01.591332 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:01.591348 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:01.591379 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:01.591415 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:01.594751 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:01.595401 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:01.595409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:01.595411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:01.595530 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:01.595852 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:01.595856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:01.595860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:01.596221 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:01.596496 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:01.596736 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 15:05:01.596928 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:05:01.597826 30590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:01.688206 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 30388
I20260925 15:05:01.702412 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 30606
I20260925 15:05:01.717119 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.703136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.22.236.2:38433)
0925 15:05:01.703790 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:01.703795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:01.703813 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:05:01.703865 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:01.703870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:01.703944 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:01.704059 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:01.705405 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.706489 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.707660 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.708338 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.708450 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:01.710848 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:01.710866 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:01.710888 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:01.710923 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:01.714454 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:01.715071 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:01.715080 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:01.715084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:01.715240 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:01.715573 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:01.715578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:01.715581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:01.715968 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:01.716280 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:01.716532 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 15:05:01.716739 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":443,"threads_started":1}
I20260925 15:05:01.717777 30735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:01.808050 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 30606
I20260925 15:05:01.821295 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 30753
I20260925 15:05:01.833554 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.822004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35354 (local address 127.22.236.3:35359)
0925 15:05:01.822205 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:01.822210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:01.822267 (+ 57us) server_negotiation.cc:407] Connection header received
0925 15:05:01.822442 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:01.822446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:01.822516 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:01.822604 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:01.823840 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.824597 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.825791 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:01.826166 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:01.826272 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:01.828656 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:01.828670 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:01.828686 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:01.828718 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:01.831072 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:01.831649 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:01.831658 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:01.831660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:01.831782 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:01.832124 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:01.832128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:01.832130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:01.832482 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:01.832773 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:01.833062 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 15:05:01.833269 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:05:01.834126 30876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:01.947794 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 30753
I20260925 15:05:01.960988 30980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.960597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34682)
0925 15:05:01.960749 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.960837 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:05:01.961880 30980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:01.961592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:45627 (local address 127.0.0.1:34688)
0925 15:05:01.961718 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:01.961782 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 15:05:01.965005 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 30084
I20260925 15:05:01.965989 30256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:02.077135 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 30084
I20260925 15:05:02.092919 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 30183
I20260925 15:05:02.093905 30446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:02.218546 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 30183
2026-09-25T15:05:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10976 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:05:02 dist-test-slave-nsrc krb5kdc[31021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:05:02 dist-test-slave-nsrc krb5kdc[31021](info): set up 2 sockets
Sep 25 15:05:02 dist-test-slave-nsrc krb5kdc[31021](info): commencing operation
krb5kdc: starting...
W20260925 15:05:03.208904 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:05:03 dist-test-slave-nsrc krb5kdc[31021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:05:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.62.keytab.
Entry for principal oryx/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.62.keytab.
I20260925 15:05:03.402558 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:37671
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.62.keytab
--principal=oryx/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:37129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:37671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:05:03.709991 31034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:03.710306 31034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:03.710438 31034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:03.717765 31034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:05:03.717900 31034 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:05:03.717968 31034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:03.718052 31034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:03.718113 31034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:05:03.718194 31034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:05:03.728036 31034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:37129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:37671
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.62.keytab
--principal=oryx/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:37671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:03.729421 31034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:03.730681 31034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:03.742097 31039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:03.742161 31040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:03.742071 31042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:03.742640 31034 server_base.cc:1034] running on GCE node
I20260925 15:05:03.743426 31034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:03.744915 31034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:03.746112 31034 hybrid_clock.cc:648] HybridClock initialized: now 1790348703746048 us; error 49 us; skew 500 ppm
I20260925 15:05:03.746421 31034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:03 dist-test-slave-nsrc krb5kdc[31021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348703, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:03.749393 31034 init.cc:377] Logged in from keytab as oryx/127.22.236.62@KRBTEST.COM (short username oryx)
I20260925 15:05:03.750718 31034 webserver.cc:460] Webserver started at http://127.22.236.62:37105/ using document root <none> and password file <none>
I20260925 15:05:03.751349 31034 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:03.751431 31034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:03.751695 31034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:03.754492 31034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "0af0892f9c264ab69bb60e77a49d3d45"
format_stamp: "Formatted at 2026-09-25 15:05:03 on dist-test-slave-nsrc"
server_key: "cf7d645c84a9804f1108fbbaaa650b98"
server_key_iv: "44bc0b4455d75ed010b72538ad245b7f"
server_key_version: "encryptionkey@0"
I20260925 15:05:03.755194 31034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "0af0892f9c264ab69bb60e77a49d3d45"
format_stamp: "Formatted at 2026-09-25 15:05:03 on dist-test-slave-nsrc"
server_key: "cf7d645c84a9804f1108fbbaaa650b98"
server_key_iv: "44bc0b4455d75ed010b72538ad245b7f"
server_key_version: "encryptionkey@0"
I20260925 15:05:03.759980 31034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:05:03.763175 31049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:03.764599 31034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:05:03.764753 31034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "0af0892f9c264ab69bb60e77a49d3d45"
format_stamp: "Formatted at 2026-09-25 15:05:03 on dist-test-slave-nsrc"
server_key: "cf7d645c84a9804f1108fbbaaa650b98"
server_key_iv: "44bc0b4455d75ed010b72538ad245b7f"
server_key_version: "encryptionkey@0"
I20260925 15:05:03.764978 31034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:03.784571 31034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:03.788273 31034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:03.788506 31034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:03.807606 31034 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:37671
I20260925 15:05:03.807626 31110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:37671 every 8 connection(s)
I20260925 15:05:03.808848 31034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:05:03.809988 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 31034
I20260925 15:05:03.810173 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:05:03.810982 23472 external_mini_cluster.cc:1428] Setting key e5574e76ae83aa653b22d190804f21b2
I20260925 15:05:03.813355 31111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:05:03 dist-test-slave-nsrc krb5kdc[31021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
I20260925 15:05:03.825328 31111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45: Bootstrap starting.
I20260925 15:05:03.829317 31111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:03.830516 31111 log.cc:824] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45: Log is configured to *not* fsync() on all Append() calls
I20260925 15:05:03.833590 31114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:03.812937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.22.236.62:37671)
0925 15:05:03.813654 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:03.813666 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:03.813706 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:05:03.815216 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:03.815236 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:03.815610 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:03.816003 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:03.817631 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:03.818951 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:03.820634 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:03.821263 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:03.821384 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:03.825696 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:03.825706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:03.825725 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:03.825771 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:03.829472 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:03.830063 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:03.830071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:03.830079 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:03.830281 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:03.830622 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:03.830628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:03.830630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:03.831256 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:03.831481 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:03.831926 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 15:05:03.832363 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":330,"threads_started":1}
I20260925 15:05:03.833690 31111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45: No bootstrap required, opened a new log
I20260925 15:05:03.841400 31111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af0892f9c264ab69bb60e77a49d3d45" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37671 } }
I20260925 15:05:03.841709 31111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:03.841781 31111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af0892f9c264ab69bb60e77a49d3d45, State: Initialized, Role: FOLLOWER
I20260925 15:05:03.842267 31111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af0892f9c264ab69bb60e77a49d3d45" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37671 } }
I20260925 15:05:03.842463 31111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:05:03.842555 31111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:05:03.842674 31111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:03.844686 31111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af0892f9c264ab69bb60e77a49d3d45" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37671 } }
I20260925 15:05:03.845245 31111 leader_election.cc:302] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0af0892f9c264ab69bb60e77a49d3d45; no voters:
I20260925 15:05:03.845748 31111 leader_election.cc:288] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:05:03.845861 31116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:05:03.846467 31116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [term 1 LEADER]: Becoming Leader. State: Replica: 0af0892f9c264ab69bb60e77a49d3d45, State: Running, Role: LEADER
I20260925 15:05:03.846987 31116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af0892f9c264ab69bb60e77a49d3d45" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37671 } }
I20260925 15:05:03.847503 31111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:05:03.849921 31117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0af0892f9c264ab69bb60e77a49d3d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af0892f9c264ab69bb60e77a49d3d45" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37671 } } }
I20260925 15:05:03.850273 31117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [sys.catalog]: This master's current role is: LEADER
I20260925 15:05:03.849686 31118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0af0892f9c264ab69bb60e77a49d3d45. Latest consensus state: current_term: 1 leader_uuid: "0af0892f9c264ab69bb60e77a49d3d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af0892f9c264ab69bb60e77a49d3d45" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37671 } } }
I20260925 15:05:03.851068 31118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45 [sys.catalog]: This master's current role is: LEADER
I20260925 15:05:03.856174 31126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:05:03.860915 31126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:05:03.868983 31126 catalog_manager.cc:1347] Generated new cluster ID: 694c03757370475d8572ff38f12e5acd
I20260925 15:05:03.869081 31126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:05:03.885072 31126 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:05:03.886278 31126 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:05:03.896370 31126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45: Generated new TSK 0
I20260925 15:05:03.897116 31126 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.1.keytab.
Entry for principal oryx/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.1.keytab.
I20260925 15:05:03.954010 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37671
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.1.keytab
--principal=oryx/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:37129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:05:04.266219 31137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:04.266522 31137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:04.266655 31137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:04.274160 31137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:04.274279 31137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:04.274399 31137 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:05:04.274510 31137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:05:04.284118 31137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:37129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.1.keytab
--principal=oryx/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.236.62:37671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:04.285357 31137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:04.286605 31137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:04.298586 31142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:04.298554 31143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:04.298511 31145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:04.299168 31137 server_base.cc:1034] running on GCE node
I20260925 15:05:04.299817 31137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:04.300715 31137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:04.301927 31137 hybrid_clock.cc:648] HybridClock initialized: now 1790348704301869 us; error 30 us; skew 500 ppm
I20260925 15:05:04.302204 31137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:04 dist-test-slave-nsrc krb5kdc[31021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348704, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:04.305119 31137 init.cc:377] Logged in from keytab as oryx/127.22.236.1@KRBTEST.COM (short username oryx)
I20260925 15:05:04.306402 31137 webserver.cc:460] Webserver started at http://127.22.236.1:40107/ using document root <none> and password file <none>
I20260925 15:05:04.307046 31137 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:04.307139 31137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:04.307425 31137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:04.310177 31137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "148107113cf14e118b4170de6bec675a"
format_stamp: "Formatted at 2026-09-25 15:05:04 on dist-test-slave-nsrc"
server_key: "e816ff33cef53f521b281d25583b3155"
server_key_iv: "f2a4514e4240aa8cd132166409547b75"
server_key_version: "encryptionkey@0"
I20260925 15:05:04.310995 31137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "148107113cf14e118b4170de6bec675a"
format_stamp: "Formatted at 2026-09-25 15:05:04 on dist-test-slave-nsrc"
server_key: "e816ff33cef53f521b281d25583b3155"
server_key_iv: "f2a4514e4240aa8cd132166409547b75"
server_key_version: "encryptionkey@0"
I20260925 15:05:04.315693 31137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 15:05:04.319047 31152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:04.320359 31137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:05:04.320497 31137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "148107113cf14e118b4170de6bec675a"
format_stamp: "Formatted at 2026-09-25 15:05:04 on dist-test-slave-nsrc"
server_key: "e816ff33cef53f521b281d25583b3155"
server_key_iv: "f2a4514e4240aa8cd132166409547b75"
server_key_version: "encryptionkey@0"
I20260925 15:05:04.320688 31137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:04.334709 31137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:04.338668 31137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:04.338896 31137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:04.341313 31137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:04.341390 31137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:04.341545 31137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:04.341648 31137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 25 15:05:04 dist-test-slave-nsrc krb5kdc[31021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348704, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
I20260925 15:05:04.363193 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:04.341733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:34331 (local address 127.22.236.62:37671)
0925 15:05:04.342460 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:04.342466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:04.343369 (+ 903us) server_negotiation.cc:407] Connection header received
0925 15:05:04.345353 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:04.345359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:04.345449 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:04.345544 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:04.347629 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.348633 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.351009 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.351616 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.351724 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:04.356456 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:04.356463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:04.356468 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:04.356510 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:04.359552 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:04.360515 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:04.360520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:04.360524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:04.360669 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:04.361187 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:04.361194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:04.361198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:04.361569 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:04.361723 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:04.362500 (+ 777us) server_negotiation.cc:299] Negotiation successful
0925 15:05:04.362760 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":495,"threads_started":1}
I20260925 15:05:04.363938 31162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:04.342322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37671 (local address 127.22.236.1:34331)
0925 15:05:04.343196 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:04.343244 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:04.345105 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:04.345803 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:04.345816 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:04.346566 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:04.347407 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:04.347434 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.348791 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.348800 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:04.350880 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:04.350893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.351857 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.351865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:04.352167 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:04.353048 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:04.353078 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:04.356219 (+ 3141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:04.359691 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:04.359706 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:04.359724 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:04.360356 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:04.360772 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:04.360777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:04.360781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:04.361043 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:04.361697 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:04.361703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:04.362147 (+ 444us) client_negotiation.cc:769] Sending connection context
0925 15:05:04.362373 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 15:05:04.362764 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":342,"threads_started":1}
I20260925 15:05:04.382795 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:04.369327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:49483 (local address 127.22.236.62:37671)
0925 15:05:04.369473 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:04.369478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:04.370288 (+ 810us) server_negotiation.cc:407] Connection header received
0925 15:05:04.370443 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:04.370447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:04.370505 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:04.370589 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:04.371893 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.373049 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.374429 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.374913 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.375040 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:04.377372 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:04.377377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:04.377382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:04.377415 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:04.380007 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:04.380805 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:04.380810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:04.380814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:04.380969 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:04.381402 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:04.381407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:04.381411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:04.381673 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:04.381839 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:04.382365 (+ 526us) server_negotiation.cc:299] Negotiation successful
0925 15:05:04.382571 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:05:04.383085 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:04.369424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37671 (local address 127.22.236.1:49483)
0925 15:05:04.370178 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:04.370200 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:04.370344 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:04.370646 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:04.370651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:04.371243 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:04.371689 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:04.371704 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.373205 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.373209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:04.374270 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:04.374282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.375119 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.375126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:04.375325 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:04.376012 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:04.376037 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:04.377160 (+ 1123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:04.380135 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:04.380162 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:04.380167 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:04.380697 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:04.381113 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:04.381119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:04.381126 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:04.381279 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:04.381811 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:04.381816 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:04.382067 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 15:05:04.382250 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 15:05:04.382620 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":359,"threads_started":1}
I20260925 15:05:04.390446 31137 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:33199
I20260925 15:05:04.390491 31272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:33199 every 8 connection(s)
I20260925 15:05:04.391548 31137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:05:04.393471 31273 heartbeater.cc:347] Connected to a master server at 127.22.236.62:37671
I20260925 15:05:04.393679 31273 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:04.394326 31273 heartbeater.cc:511] Master 127.22.236.62:37671 requested a full tablet report, sending...
I20260925 15:05:04.395006 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 31137
I20260925 15:05:04.395186 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:05:04.395807 23472 external_mini_cluster.cc:1428] Setting key c23cd519e4df15783102370f72111b7f
I20260925 15:05:04.396337 31066 ts_manager.cc:194] Registered new tserver with Master: 148107113cf14e118b4170de6bec675a (127.22.236.1:33199)
I20260925 15:05:04.397810 31066 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:34331
WARNING: no policy specified for oryx/127.22.236.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.236.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.2.keytab.
Entry for principal oryx/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.2.keytab.
I20260925 15:05:04.435343 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37671
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.2.keytab
--principal=oryx/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:37129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:05:04.743431 31277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:04.743700 31277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:04.743832 31277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:04.751000 31277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:04.751118 31277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:04.751237 31277 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:05:04.751348 31277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:05:04.760993 31277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:37129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.2.keytab
--principal=oryx/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.236.62:37671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:04.762172 31277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:04.763442 31277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:04.775621 31283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:04.775669 31285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:04.775782 31282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:04.776279 31277 server_base.cc:1034] running on GCE node
I20260925 15:05:04.776871 31277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:04.777719 31277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:04.778923 31277 hybrid_clock.cc:648] HybridClock initialized: now 1790348704778861 us; error 41 us; skew 500 ppm
I20260925 15:05:04.779193 31277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:04 dist-test-slave-nsrc krb5kdc[31021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348704, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:04.782094 31277 init.cc:377] Logged in from keytab as oryx/127.22.236.2@KRBTEST.COM (short username oryx)
I20260925 15:05:04.783380 31277 webserver.cc:460] Webserver started at http://127.22.236.2:34767/ using document root <none> and password file <none>
I20260925 15:05:04.783985 31277 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:04.784070 31277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:04.784409 31277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:04.787098 31277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "7062aeb28dd74e0e9517946ec13a13ea"
format_stamp: "Formatted at 2026-09-25 15:05:04 on dist-test-slave-nsrc"
server_key: "f657b6048854ed1037837d5e623c0bd1"
server_key_iv: "037bb0d36ed805535f43cdb46427e480"
server_key_version: "encryptionkey@0"
I20260925 15:05:04.787811 31277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "7062aeb28dd74e0e9517946ec13a13ea"
format_stamp: "Formatted at 2026-09-25 15:05:04 on dist-test-slave-nsrc"
server_key: "f657b6048854ed1037837d5e623c0bd1"
server_key_iv: "037bb0d36ed805535f43cdb46427e480"
server_key_version: "encryptionkey@0"
I20260925 15:05:04.792634 31277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:05:04.795765 31292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:04.797015 31277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:05:04.797132 31277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "7062aeb28dd74e0e9517946ec13a13ea"
format_stamp: "Formatted at 2026-09-25 15:05:04 on dist-test-slave-nsrc"
server_key: "f657b6048854ed1037837d5e623c0bd1"
server_key_iv: "037bb0d36ed805535f43cdb46427e480"
server_key_version: "encryptionkey@0"
I20260925 15:05:04.797299 31277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:04.812747 31277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:04.816606 31277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:04.816830 31277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:04.819391 31277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:04.819471 31277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:04.819635 31277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:04.819698 31277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:05:04 dist-test-slave-nsrc krb5kdc[31021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348704, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
I20260925 15:05:04.844611 31301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:04.820787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37671 (local address 127.22.236.2:60257)
0925 15:05:04.821636 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:04.821684 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:04.824188 (+ 2504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:04.825244 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:04.825255 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:04.826023 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:04.826881 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:04.826907 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.828183 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.828192 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:04.830549 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:04.830564 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.831430 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.831438 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:04.831752 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:04.832700 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:04.832733 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:04.835874 (+ 3141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:04.839484 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:04.839492 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:04.839517 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:04.840178 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:04.840594 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:04.840599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:04.840603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:04.840984 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:04.841665 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:04.841672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:04.842129 (+ 457us) client_negotiation.cc:769] Sending connection context
0925 15:05:04.842421 (+ 292us) client_negotiation.cc:240] Negotiation successful
0925 15:05:04.842977 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":318,"threads_started":1}
I20260925 15:05:04.844851 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:04.820240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:60257 (local address 127.22.236.62:37671)
0925 15:05:04.820430 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:04.820437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:04.821810 (+ 1373us) server_negotiation.cc:407] Connection header received
0925 15:05:04.824569 (+ 2759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:04.824576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:04.824661 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:04.824778 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:04.827048 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.828011 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.830707 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.831178 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.831307 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:04.836084 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:04.836092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:04.836096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:04.836155 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:04.839314 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:04.840325 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:04.840331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:04.840334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:04.840487 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:04.841141 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:04.841144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:04.841146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:04.841466 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:04.841610 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:04.844317 (+ 2707us) server_negotiation.cc:299] Negotiation successful
0925 15:05:04.844531 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:05:04.862304 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:04.850424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:52237 (local address 127.22.236.62:37671)
0925 15:05:04.850576 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:04.850582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:04.850685 (+ 103us) server_negotiation.cc:407] Connection header received
0925 15:05:04.850853 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:04.850859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:04.850919 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:04.850991 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:04.851861 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.852742 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.853996 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.854399 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.854498 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:04.856554 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:04.856559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:04.856562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:04.856589 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:04.859505 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:04.860428 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:04.860434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:04.860438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:04.860586 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:04.860938 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:04.860940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:04.860942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:04.861295 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:04.861461 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:04.861922 (+ 461us) server_negotiation.cc:299] Negotiation successful
0925 15:05:04.862082 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:05:04.862687 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:04.849947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37671 (local address 127.22.236.2:52237)
0925 15:05:04.850605 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:04.850621 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:04.850745 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:04.851024 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:04.851028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:04.851437 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:04.851737 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:04.851746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.852874 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.852879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:04.853872 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:04.853885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:04.854560 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:04.854567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:04.854711 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:04.855333 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:04.855349 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:04.856309 (+ 960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:04.859657 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:04.859662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:04.859670 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:04.860301 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:04.860698 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:04.860702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:04.860705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:04.860847 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:04.861422 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:04.861436 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:04.861667 (+ 231us) client_negotiation.cc:769] Sending connection context
0925 15:05:04.861825 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 15:05:04.862166 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":527,"threads_started":1}
I20260925 15:05:04.866698 31277 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:36833
I20260925 15:05:04.866720 31411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:36833 every 8 connection(s)
I20260925 15:05:04.867835 31277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:05:04.869660 31412 heartbeater.cc:347] Connected to a master server at 127.22.236.62:37671
I20260925 15:05:04.869853 31412 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:04.870445 31412 heartbeater.cc:511] Master 127.22.236.62:37671 requested a full tablet report, sending...
I20260925 15:05:04.871599 31066 ts_manager.cc:194] Registered new tserver with Master: 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833)
I20260925 15:05:04.872401 31066 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:60257
I20260925 15:05:04.872784 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 31277
I20260925 15:05:04.872967 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:05:04.873396 23472 external_mini_cluster.cc:1428] Setting key dc7d9c2ea27ec73a1da95774481621fb
WARNING: no policy specified for oryx/127.22.236.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.236.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.3.keytab.
Entry for principal oryx/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.3.keytab.
I20260925 15:05:04.913398 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37671
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.3.keytab
--principal=oryx/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:37129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:05:05.221019 31416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:05.221271 31416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:05.221362 31416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:05.228385 31416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:05.228490 31416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:05.228578 31416 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:05:05.228657 31416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:05:05.238008 31416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:37129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.3.keytab
--principal=oryx/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.236.62:37671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:05.239188 31416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:05.240461 31416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:05.252585 31421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:05.252611 31424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:05.252651 31422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:05.253381 31416 server_base.cc:1034] running on GCE node
I20260925 15:05:05.253916 31416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:05.254693 31416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:05.255898 31416 hybrid_clock.cc:648] HybridClock initialized: now 1790348705255843 us; error 26 us; skew 500 ppm
I20260925 15:05:05.256172 31416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:05 dist-test-slave-nsrc krb5kdc[31021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348705, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:05.259028 31416 init.cc:377] Logged in from keytab as oryx/127.22.236.3@KRBTEST.COM (short username oryx)
I20260925 15:05:05.260416 31416 webserver.cc:460] Webserver started at http://127.22.236.3:40717/ using document root <none> and password file <none>
I20260925 15:05:05.261031 31416 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:05.261111 31416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:05.261363 31416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:05.264057 31416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "5d1132db8fd64678a3f5847950411e6d"
format_stamp: "Formatted at 2026-09-25 15:05:05 on dist-test-slave-nsrc"
server_key: "96d6a1c17e3afbcedfc5ff71e1a94766"
server_key_iv: "5c57714924bcd4d0035cb0f508755c10"
server_key_version: "encryptionkey@0"
I20260925 15:05:05.264806 31416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "5d1132db8fd64678a3f5847950411e6d"
format_stamp: "Formatted at 2026-09-25 15:05:05 on dist-test-slave-nsrc"
server_key: "96d6a1c17e3afbcedfc5ff71e1a94766"
server_key_iv: "5c57714924bcd4d0035cb0f508755c10"
server_key_version: "encryptionkey@0"
I20260925 15:05:05.269765 31416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 15:05:05.272931 31431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:05.274113 31416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:05:05.274231 31416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "5d1132db8fd64678a3f5847950411e6d"
format_stamp: "Formatted at 2026-09-25 15:05:05 on dist-test-slave-nsrc"
server_key: "96d6a1c17e3afbcedfc5ff71e1a94766"
server_key_iv: "5c57714924bcd4d0035cb0f508755c10"
server_key_version: "encryptionkey@0"
I20260925 15:05:05.274391 31416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:05.289801 31416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:05.293615 31416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:05.293835 31416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:05.296361 31416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:05.296527 31416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:05.296665 31416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:05.296729 31416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:05:05 dist-test-slave-nsrc krb5kdc[31021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348705, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
I20260925 15:05:05.318451 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:05.296952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:49469 (local address 127.22.236.62:37671)
0925 15:05:05.297152 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:05.297158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:05.298548 (+ 1390us) server_negotiation.cc:407] Connection header received
0925 15:05:05.300861 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:05.300868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:05.300952 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:05.301090 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:05.303173 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:05.304342 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:05.306469 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:05.306964 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:05.307090 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:05.311763 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:05.311770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:05.311775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:05.311807 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:05.315039 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:05.315738 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:05.315743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:05.315748 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:05.315920 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:05.316400 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:05.316406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:05.316410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:05.316766 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:05.316942 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:05.317852 (+ 910us) server_negotiation.cc:299] Negotiation successful
0925 15:05:05.318078 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:05:05.319572 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:05.297575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37671 (local address 127.22.236.3:49469)
0925 15:05:05.298372 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:05.298425 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:05.300505 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:05.301317 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:05.301328 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:05.302145 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:05.302985 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:05.303014 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:05.304548 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:05.304554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:05.306342 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:05.306353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:05.307367 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:05.307373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:05.307632 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:05.308424 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:05.308461 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:05.311567 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:05.315182 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:05.315191 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:05.315210 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:05.315634 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:05.316021 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:05.316024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:05.316026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:05.316290 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:05.316924 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:05.316932 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:05.317439 (+ 507us) client_negotiation.cc:769] Sending connection context
0925 15:05:05.317701 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 15:05:05.318269 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":321,"threads_started":1}
I20260925 15:05:05.338279 31416 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:43511
I20260925 15:05:05.338299 31550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:43511 every 8 connection(s)
I20260925 15:05:05.339636 31416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:05:05.340819 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 31416
I20260925 15:05:05.340981 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:05:05.341398 23472 external_mini_cluster.cc:1428] Setting key bcfc8beb5410d1e4f5efd55bcb836d4c
I20260925 15:05:05.343380 31551 heartbeater.cc:347] Connected to a master server at 127.22.236.62:37671
I20260925 15:05:05.343611 31551 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:05.344409 31551 heartbeater.cc:511] Master 127.22.236.62:37671 requested a full tablet report, sending...
I20260925 15:05:05.345408 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:05.330886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37671 (local address 127.22.236.3:54925)
0925 15:05:05.331556 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:05.331577 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:05.331726 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:05.332023 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:05.332027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:05.332700 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:05.333161 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:05.333175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:05.334380 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:05.334386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:05.336226 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:05.336243 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:05.336978 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:05.336988 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:05.337152 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:05.337790 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:05.337815 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:05.338853 (+ 1038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:05.342547 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:05.342552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:05.342556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:05.343300 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:05.343732 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:05.343736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:05.343739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:05.343875 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:05.344504 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:05.344510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:05.344684 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 15:05:05.344792 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:05:05.345027 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":355,"threads_started":1}
I20260925 15:05:05.345690 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:05.330902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:54925 (local address 127.22.236.62:37671)
0925 15:05:05.331086 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:05.331092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:05.331662 (+ 570us) server_negotiation.cc:407] Connection header received
0925 15:05:05.331845 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:05.331850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:05.331909 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:05.331977 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:05.333384 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:05.334246 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:05.336368 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:05.336782 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:05.336880 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:05.339068 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:05.339075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:05.339080 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:05.339121 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:05.342293 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:05.343443 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:05.343449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:05.343452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:05.343605 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:05.344013 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:05.344018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:05.344021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:05.344389 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:05.344533 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:05.344936 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 15:05:05.345141 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:05:05.345813 31066 ts_manager.cc:194] Registered new tserver with Master: 5d1132db8fd64678a3f5847950411e6d (127.22.236.3:43511)
I20260925 15:05:05.346706 31066 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:49469
I20260925 15:05:05.356060 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:05:05 dist-test-slave-nsrc krb5kdc[31021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348705, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:05:05 dist-test-slave-nsrc krb5kdc[31021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:05 dist-test-slave-nsrc krb5kdc[31021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:05.400501 31273 heartbeater.cc:503] Master 127.22.236.62:37671 was elected leader, sending a full tablet report...
I20260925 15:05:05.874801 31412 heartbeater.cc:503] Master 127.22.236.62:37671 was elected leader, sending a full tablet report...
I20260925 15:05:06.349066 31551 heartbeater.cc:503] Master 127.22.236.62:37671 was elected leader, sending a full tablet report...
W20260925 15:05:06.353193 31559 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.957s user 0.006s sys 0.006s
W20260925 15:05:06.354247 31161 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:05.390564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42242 (local address 127.22.236.62:37671)
0925 15:05:05.390753 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:05.390758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:05.390989 (+ 231us) server_negotiation.cc:407] Connection header received
0925 15:05:05.391174 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:05.391178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:05.391250 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:05.391338 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:05.392957 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:05.393720 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:05.395426 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:05.395805 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:05.395895 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.353959 (+958064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42242: BlockingRecv error: recv got EOF from 127.0.0.1:42242 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 15:05:06 dist-test-slave-nsrc krb5kdc[31021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348705, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
I20260925 15:05:06.374624 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:06.357622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.22.236.62:37671)
0925 15:05:06.357790 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:06.357797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:06.358056 (+ 259us) server_negotiation.cc:407] Connection header received
0925 15:05:06.358214 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:06.358219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:06.358295 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:06.358374 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:06.359543 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.360279 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.361448 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.361869 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.361971 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.369716 (+ 7745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:06.369721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:06.369725 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:06.369759 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:06.372119 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:06.372788 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:06.372793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:06.372797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:06.372924 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:06.373295 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:06.373300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:06.373304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:06.373584 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:06.373858 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:06.374105 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 15:05:06.374333 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:05:06.378422 31066 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:05:06.385150 31066 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:05:06.406868 31570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:06.396745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:43511 (local address 127.0.0.1:43670)
0925 15:05:06.397455 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:06.397503 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:06.397672 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:06.398843 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:06.398850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:06.398881 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:06.399305 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:06.399316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.401343 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.401348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:06.403866 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:06.403877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.405936 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.405942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:06.406148 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.406201 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:05:06.406332 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 15:05:06.406519 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":291,"threads_started":1}
I20260925 15:05:06.406880 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:06.396927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.22.236.3:43511)
0925 15:05:06.397973 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:06.397981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:06.398008 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:05:06.398116 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:06.398125 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:06.398605 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:06.399515 (+ 910us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:06.399696 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.401202 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.404003 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.405174 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.405617 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.405819 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:06.406423 (+ 604us) server_negotiation.cc:299] Negotiation successful
0925 15:05:06.406591 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":1369,"threads_started":1}
I20260925 15:05:06.410065 31573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:06.397407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:36833 (local address 127.0.0.1:49350)
0925 15:05:06.401936 (+ 4529us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:06.401955 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:06.402065 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:06.403050 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:06.403054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:06.403074 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:06.403478 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:06.403488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.405318 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.405324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:06.407434 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:06.407446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.409301 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.409307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:06.409462 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.409492 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:05:06.409552 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:05:06.409676 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4424,"mutex_wait_us":83,"thread_start_us":4254,"threads_started":1}
I20260925 15:05:06.410511 31574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:06.397876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52542 (local address 127.22.236.1:33199)
0925 15:05:06.398333 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:06.398342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:06.402043 (+ 3701us) server_negotiation.cc:407] Connection header received
0925 15:05:06.402353 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:06.402360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:06.402529 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:06.402676 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:06.403619 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.405239 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.407876 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.409281 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.409413 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.409603 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:06.410091 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 15:05:06.410255 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":596,"threads_started":1}
I20260925 15:05:06.410917 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:06.397954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.22.236.2:36833)
0925 15:05:06.401478 (+ 3524us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:06.401486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:06.402175 (+ 689us) server_negotiation.cc:407] Connection header received
0925 15:05:06.402269 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:06.402277 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:06.402702 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:06.402862 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:06.403663 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.405157 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.407569 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.408941 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.410287 (+ 1346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.410476 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:06.410542 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 15:05:06.410704 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3370,"thread_start_us":3250,"threads_started":1}
I20260925 15:05:06.410491 31572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:06.397343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33199 (local address 127.0.0.1:52542)
0925 15:05:06.401943 (+ 4600us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:06.401956 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:06.402117 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:06.402924 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:06.402928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:06.402950 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:06.403420 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:06.403432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.405352 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.405357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:06.407727 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:06.407740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.409731 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.409737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:06.409901 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.409931 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:05:06.410029 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 15:05:06.410151 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3677,"mutex_wait_us":42,"thread_start_us":3530,"threads_started":1}
I20260925 15:05:06.410694 31481 tablet_service.cc:1467] Processing CreateTablet for tablet ee204c9dbe36435dabfedfcb5e7ce4a7 (DEFAULT_TABLE table=test-table [id=6ef8ce8fd6d04f4294a5703552ec32a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:06.411937 31481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee204c9dbe36435dabfedfcb5e7ce4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:06.414594 31203 tablet_service.cc:1467] Processing CreateTablet for tablet ee204c9dbe36435dabfedfcb5e7ce4a7 (DEFAULT_TABLE table=test-table [id=6ef8ce8fd6d04f4294a5703552ec32a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:06.415248 31342 tablet_service.cc:1467] Processing CreateTablet for tablet ee204c9dbe36435dabfedfcb5e7ce4a7 (DEFAULT_TABLE table=test-table [id=6ef8ce8fd6d04f4294a5703552ec32a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:06.415899 31203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee204c9dbe36435dabfedfcb5e7ce4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:06.416312 31342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee204c9dbe36435dabfedfcb5e7ce4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:06.426080 31576 tablet_bootstrap.cc:492] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d: Bootstrap starting.
I20260925 15:05:06.428812 31577 tablet_bootstrap.cc:492] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea: Bootstrap starting.
I20260925 15:05:06.430207 31578 tablet_bootstrap.cc:492] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a: Bootstrap starting.
I20260925 15:05:06.430305 31576 tablet_bootstrap.cc:654] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:06.431917 31577 tablet_bootstrap.cc:654] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:06.431942 31576 log.cc:824] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:05:06.433017 31577 log.cc:824] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea: Log is configured to *not* fsync() on all Append() calls
I20260925 15:05:06.433466 31578 tablet_bootstrap.cc:654] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:06.434660 31578 log.cc:824] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:05:06.435050 31576 tablet_bootstrap.cc:492] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d: No bootstrap required, opened a new log
I20260925 15:05:06.435285 31577 tablet_bootstrap.cc:492] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea: No bootstrap required, opened a new log
I20260925 15:05:06.435380 31576 ts_tablet_manager.cc:1397] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 15:05:06.435559 31577 ts_tablet_manager.cc:1397] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 15:05:06.437732 31578 tablet_bootstrap.cc:492] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a: No bootstrap required, opened a new log
I20260925 15:05:06.437976 31578 ts_tablet_manager.cc:1397] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260925 15:05:06.442436 31577 raft_consensus.cc:348] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } }
I20260925 15:05:06.442785 31577 raft_consensus.cc:374] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:06.442893 31577 raft_consensus.cc:729] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7062aeb28dd74e0e9517946ec13a13ea, State: Initialized, Role: FOLLOWER
I20260925 15:05:06.443436 31577 consensus_queue.cc:260] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } }
I20260925 15:05:06.445101 31577 ts_tablet_manager.cc:1428] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260925 15:05:06.445363 31576 raft_consensus.cc:348] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } }
I20260925 15:05:06.445758 31576 raft_consensus.cc:374] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:06.445853 31576 raft_consensus.cc:729] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1132db8fd64678a3f5847950411e6d, State: Initialized, Role: FOLLOWER
I20260925 15:05:06.446514 31576 consensus_queue.cc:260] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } }
I20260925 15:05:06.448589 31578 raft_consensus.cc:348] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } }
I20260925 15:05:06.448999 31578 raft_consensus.cc:374] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:06.449146 31578 raft_consensus.cc:729] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148107113cf14e118b4170de6bec675a, State: Initialized, Role: FOLLOWER
I20260925 15:05:06.449772 31578 consensus_queue.cc:260] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } }
I20260925 15:05:06.451231 31576 ts_tablet_manager.cc:1428] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260925 15:05:06.451314 31578 ts_tablet_manager.cc:1428] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260925 15:05:06.546259 31584 raft_consensus.cc:482] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:05:06.546550 31584 raft_consensus.cc:504] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } }
I20260925 15:05:06.548252 31584 leader_election.cc:288] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d1132db8fd64678a3f5847950411e6d (127.22.236.3:43511), 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833)
I20260925 15:05:06.556593 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:06.548597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:57063 (local address 127.22.236.3:43511)
0925 15:05:06.548799 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:06.548805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:06.549535 (+ 730us) server_negotiation.cc:407] Connection header received
0925 15:05:06.549769 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:06.549774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:06.549846 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:06.549961 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:06.550644 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.551663 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.554379 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.555603 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.555748 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.555819 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:06.556227 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 15:05:06.556352 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:05:06.556594 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:06.548764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:45019 (local address 127.22.236.2:36833)
0925 15:05:06.548947 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:06.548953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:06.549431 (+ 478us) server_negotiation.cc:407] Connection header received
0925 15:05:06.549690 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:06.549696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:06.549765 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:06.549852 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:06.550515 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.551596 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.554379 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.555603 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.555748 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.555819 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:06.556227 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 15:05:06.556352 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:05:06.556795 31586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:06.548520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:36833 (local address 127.22.236.1:45019)
0925 15:05:06.549328 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:06.549348 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:06.549549 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:06.549881 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:06.549885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:06.549909 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:06.550357 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:06.550370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.551700 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.551703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:06.554237 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:06.554249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.555811 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.555818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:06.556060 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.556108 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:05:06.556205 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:05:06.556336 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":196,"thread_start_us":573,"threads_started":1}
I20260925 15:05:06.556847 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:06.548532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:43511 (local address 127.22.236.1:57063)
0925 15:05:06.549435 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:06.549450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:06.549589 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:06.550053 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:06.550057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:06.550082 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:06.550496 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:06.550506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.551773 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.551777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:06.554235 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:06.554249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.555847 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.555853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:06.556068 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.556101 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:05:06.556245 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 15:05:06.556346 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":191,"spinlock_wait_cycles":21760,"thread_start_us":657,"threads_started":1}
I20260925 15:05:06.557721 31362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee204c9dbe36435dabfedfcb5e7ce4a7" candidate_uuid: "148107113cf14e118b4170de6bec675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7062aeb28dd74e0e9517946ec13a13ea" is_pre_election: true
I20260925 15:05:06.558138 31362 raft_consensus.cc:2454] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148107113cf14e118b4170de6bec675a in term 0.
I20260925 15:05:06.558722 31155 leader_election.cc:302] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148107113cf14e118b4170de6bec675a, 7062aeb28dd74e0e9517946ec13a13ea; no voters:
I20260925 15:05:06.558851 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee204c9dbe36435dabfedfcb5e7ce4a7" candidate_uuid: "148107113cf14e118b4170de6bec675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1132db8fd64678a3f5847950411e6d" is_pre_election: true
I20260925 15:05:06.559116 31584 raft_consensus.cc:2781] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:05:06.559243 31584 raft_consensus.cc:482] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:05:06.559330 31584 raft_consensus.cc:3037] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:06.559300 31501 raft_consensus.cc:2454] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148107113cf14e118b4170de6bec675a in term 0.
I20260925 15:05:06.561560 31584 raft_consensus.cc:504] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } }
I20260925 15:05:06.562112 31584 leader_election.cc:288] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [CANDIDATE]: Term 1 election: Requested vote from peers 5d1132db8fd64678a3f5847950411e6d (127.22.236.3:43511), 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833)
I20260925 15:05:06.562583 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee204c9dbe36435dabfedfcb5e7ce4a7" candidate_uuid: "148107113cf14e118b4170de6bec675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1132db8fd64678a3f5847950411e6d"
I20260925 15:05:06.562583 31362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee204c9dbe36435dabfedfcb5e7ce4a7" candidate_uuid: "148107113cf14e118b4170de6bec675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7062aeb28dd74e0e9517946ec13a13ea"
I20260925 15:05:06.562783 31362 raft_consensus.cc:3037] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:06.562783 31501 raft_consensus.cc:3037] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:06.565873 31362 raft_consensus.cc:2454] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148107113cf14e118b4170de6bec675a in term 1.
I20260925 15:05:06.565873 31501 raft_consensus.cc:2454] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148107113cf14e118b4170de6bec675a in term 1.
I20260925 15:05:06.566283 31153 leader_election.cc:302] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 148107113cf14e118b4170de6bec675a, 5d1132db8fd64678a3f5847950411e6d, 7062aeb28dd74e0e9517946ec13a13ea; no voters:
I20260925 15:05:06.566582 31584 raft_consensus.cc:2781] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:05:06.566983 31584 raft_consensus.cc:686] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [term 1 LEADER]: Becoming Leader. State: Replica: 148107113cf14e118b4170de6bec675a, State: Running, Role: LEADER
I20260925 15:05:06.567457 31584 consensus_queue.cc:237] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } }
I20260925 15:05:06.571619 31066 catalog_manager.cc:5495] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a reported cstate change: term changed from 0 to 1, leader changed from <none> to 148107113cf14e118b4170de6bec675a (127.22.236.1). New cstate: current_term: 1 leader_uuid: "148107113cf14e118b4170de6bec675a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } health_report { overall_health: UNKNOWN } } }
W20260925 15:05:06.593894 31552 tablet.cc:2369] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:05:06.597728 31574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:06.592787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52550 (local address 127.22.236.1:33199)
0925 15:05:06.592979 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:06.592985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:06.593072 (+ 87us) server_negotiation.cc:407] Connection header received
0925 15:05:06.593279 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:06.593283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:06.593363 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:06.593487 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:05:06.593830 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.594556 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.595810 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.596213 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.596314 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.596702 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:05:06.597212 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:05:06.597329 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:06.597415 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 15:05:06.597491 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:05:06.606186 31362 raft_consensus.cc:1261] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Refusing update from remote peer 148107113cf14e118b4170de6bec675a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:05:06.606186 31501 raft_consensus.cc:1261] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Refusing update from remote peer 148107113cf14e118b4170de6bec675a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:05:06.607175 31587 consensus_queue.cc:1035] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:05:06.607478 31584 consensus_queue.cc:1035] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:05:06.618472 31590 mvcc.cc:205] Tried to move back new op lower bound from 7333268302249209856 to 7333268302104199168. Current Snapshot: MvccSnapshot[applied={T|T < 7333268302249209856}]
I20260925 15:05:06.619160 31591 mvcc.cc:205] Tried to move back new op lower bound from 7333268302249209856 to 7333268302104199168. Current Snapshot: MvccSnapshot[applied={T|T < 7333268302249209856}]
I20260925 15:05:06.635113 31066 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:05:06.635381 31066 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:05:06.638710 31066 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45: Sending DeleteTablet for 3 replicas of tablet ee204c9dbe36435dabfedfcb5e7ce4a7
I20260925 15:05:06.639761 31203 tablet_service.cc:1514] Processing DeleteTablet for tablet ee204c9dbe36435dabfedfcb5e7ce4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:06 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:52542
I20260925 15:05:06.639761 31481 tablet_service.cc:1514] Processing DeleteTablet for tablet ee204c9dbe36435dabfedfcb5e7ce4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:06 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:43670
I20260925 15:05:06.639871 31342 tablet_service.cc:1514] Processing DeleteTablet for tablet ee204c9dbe36435dabfedfcb5e7ce4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:06 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:49350
I20260925 15:05:06.641357 31066 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:05:06.642056 31066 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:05:06.653239 31341 tablet_service.cc:1467] Processing CreateTablet for tablet d6d7e36e93f240bbab198931f390ecb2 (DEFAULT_TABLE table=test-table [id=ab0d476a772c426c8828f7533c127bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:06.653836 31341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d7e36e93f240bbab198931f390ecb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:06.656564 31600 tablet_replica.cc:330] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d: stopping tablet replica
I20260925 15:05:06.657789 31481 tablet_service.cc:1467] Processing CreateTablet for tablet d6d7e36e93f240bbab198931f390ecb2 (DEFAULT_TABLE table=test-table [id=ab0d476a772c426c8828f7533c127bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:06.657869 31600 raft_consensus.cc:2229] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:05:06.658247 31481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d7e36e93f240bbab198931f390ecb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:06.658416 31600 raft_consensus.cc:2258] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:06.658749 31602 tablet_replica.cc:330] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea: stopping tablet replica
I20260925 15:05:06.658663 31202 tablet_service.cc:1467] Processing CreateTablet for tablet d6d7e36e93f240bbab198931f390ecb2 (DEFAULT_TABLE table=test-table [id=ab0d476a772c426c8828f7533c127bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:06.659165 31202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d7e36e93f240bbab198931f390ecb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:06.659281 31602 raft_consensus.cc:2229] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:05:06.659765 31602 raft_consensus.cc:2258] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:06.660773 31600 ts_tablet_manager.cc:1905] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:06.666051 31602 ts_tablet_manager.cc:1905] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:06.667605 31600 ts_tablet_manager.cc:1918] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:05:06.667771 31600 log.cc:1196] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/ee204c9dbe36435dabfedfcb5e7ce4a7
I20260925 15:05:06.667884 31576 tablet_bootstrap.cc:492] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d: Bootstrap starting.
I20260925 15:05:06.668351 31600 ts_tablet_manager.cc:1939] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 5d1132db8fd64678a3f5847950411e6d: Deleting consensus metadata
I20260925 15:05:06.669480 31576 tablet_bootstrap.cc:654] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:06.670131 31050 catalog_manager.cc:4841] TS 5d1132db8fd64678a3f5847950411e6d (127.22.236.3:43511): tablet ee204c9dbe36435dabfedfcb5e7ce4a7 (table test-table [id=6ef8ce8fd6d04f4294a5703552ec32a6]) successfully deleted
I20260925 15:05:06.671205 31577 tablet_bootstrap.cc:492] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea: Bootstrap starting.
I20260925 15:05:06.671633 31578 tablet_bootstrap.cc:492] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a: Bootstrap starting.
I20260925 15:05:06.672732 31577 tablet_bootstrap.cc:654] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:06.673398 31578 tablet_bootstrap.cc:654] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:06.658684 31601 tablet_replica.cc:330] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a: stopping tablet replica
I20260925 15:05:06.679003 31601 raft_consensus.cc:2229] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:05:06.680835 31601 raft_consensus.cc:2258] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:06.682914 31601 ts_tablet_manager.cc:1905] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:06.683189 31602 ts_tablet_manager.cc:1918] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:05:06.683349 31602 log.cc:1196] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/ee204c9dbe36435dabfedfcb5e7ce4a7
I20260925 15:05:06.683866 31602 ts_tablet_manager.cc:1939] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 7062aeb28dd74e0e9517946ec13a13ea: Deleting consensus metadata
I20260925 15:05:06.685120 31578 tablet_bootstrap.cc:492] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a: No bootstrap required, opened a new log
I20260925 15:05:06.685288 31578 ts_tablet_manager.cc:1397] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260925 15:05:06.686028 31052 catalog_manager.cc:4841] TS 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833): tablet ee204c9dbe36435dabfedfcb5e7ce4a7 (table test-table [id=6ef8ce8fd6d04f4294a5703552ec32a6]) successfully deleted
I20260925 15:05:06.686097 31578 raft_consensus.cc:348] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.686309 31578 raft_consensus.cc:374] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:06.686417 31578 raft_consensus.cc:729] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148107113cf14e118b4170de6bec675a, State: Initialized, Role: FOLLOWER
I20260925 15:05:06.686679 31578 consensus_queue.cc:260] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.687515 31577 tablet_bootstrap.cc:492] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea: No bootstrap required, opened a new log
I20260925 15:05:06.687701 31577 ts_tablet_manager.cc:1397] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20260925 15:05:06.688052 31576 tablet_bootstrap.cc:492] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d: No bootstrap required, opened a new log
I20260925 15:05:06.688251 31576 ts_tablet_manager.cc:1397] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.000s
I20260925 15:05:06.688673 31577 raft_consensus.cc:348] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.688943 31577 raft_consensus.cc:374] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:06.689085 31577 raft_consensus.cc:729] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7062aeb28dd74e0e9517946ec13a13ea, State: Initialized, Role: FOLLOWER
I20260925 15:05:06.689010 31576 raft_consensus.cc:348] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.689217 31576 raft_consensus.cc:374] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:06.689318 31576 raft_consensus.cc:729] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1132db8fd64678a3f5847950411e6d, State: Initialized, Role: FOLLOWER
I20260925 15:05:06.689540 31577 consensus_queue.cc:260] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.689556 31576 consensus_queue.cc:260] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.690120 31576 ts_tablet_manager.cc:1428] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:05:06.690363 31578 ts_tablet_manager.cc:1428] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 15:05:06.691082 31577 ts_tablet_manager.cc:1428] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 15:05:06.696343 31601 ts_tablet_manager.cc:1918] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:05:06.696532 31601 log.cc:1196] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/ee204c9dbe36435dabfedfcb5e7ce4a7
I20260925 15:05:06.697005 31601 ts_tablet_manager.cc:1939] T ee204c9dbe36435dabfedfcb5e7ce4a7 P 148107113cf14e118b4170de6bec675a: Deleting consensus metadata
I20260925 15:05:06.698455 31053 catalog_manager.cc:4841] TS 148107113cf14e118b4170de6bec675a (127.22.236.1:33199): tablet ee204c9dbe36435dabfedfcb5e7ce4a7 (table test-table [id=6ef8ce8fd6d04f4294a5703552ec32a6]) successfully deleted
I20260925 15:05:06.782253 31584 raft_consensus.cc:482] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:05:06.782467 31584 raft_consensus.cc:504] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.783106 31584 leader_election.cc:288] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833), 5d1132db8fd64678a3f5847950411e6d (127.22.236.3:43511)
I20260925 15:05:06.783624 31362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6d7e36e93f240bbab198931f390ecb2" candidate_uuid: "148107113cf14e118b4170de6bec675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7062aeb28dd74e0e9517946ec13a13ea" is_pre_election: true
I20260925 15:05:06.783653 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6d7e36e93f240bbab198931f390ecb2" candidate_uuid: "148107113cf14e118b4170de6bec675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1132db8fd64678a3f5847950411e6d" is_pre_election: true
I20260925 15:05:06.783843 31362 raft_consensus.cc:2454] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148107113cf14e118b4170de6bec675a in term 0.
I20260925 15:05:06.783907 31501 raft_consensus.cc:2454] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148107113cf14e118b4170de6bec675a in term 0.
I20260925 15:05:06.784288 31155 leader_election.cc:302] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148107113cf14e118b4170de6bec675a, 7062aeb28dd74e0e9517946ec13a13ea; no voters:
I20260925 15:05:06.784560 31584 raft_consensus.cc:2781] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:05:06.784669 31584 raft_consensus.cc:482] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:05:06.784767 31584 raft_consensus.cc:3037] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:06.786873 31584 raft_consensus.cc:504] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.787374 31584 leader_election.cc:288] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [CANDIDATE]: Term 1 election: Requested vote from peers 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833), 5d1132db8fd64678a3f5847950411e6d (127.22.236.3:43511)
I20260925 15:05:06.787837 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6d7e36e93f240bbab198931f390ecb2" candidate_uuid: "148107113cf14e118b4170de6bec675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1132db8fd64678a3f5847950411e6d"
I20260925 15:05:06.787848 31362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6d7e36e93f240bbab198931f390ecb2" candidate_uuid: "148107113cf14e118b4170de6bec675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7062aeb28dd74e0e9517946ec13a13ea"
I20260925 15:05:06.788007 31501 raft_consensus.cc:3037] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:06.788040 31362 raft_consensus.cc:3037] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:06.790325 31501 raft_consensus.cc:2454] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148107113cf14e118b4170de6bec675a in term 1.
I20260925 15:05:06.790683 31153 leader_election.cc:302] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148107113cf14e118b4170de6bec675a, 5d1132db8fd64678a3f5847950411e6d; no voters:
I20260925 15:05:06.790942 31584 raft_consensus.cc:2781] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:05:06.790931 31362 raft_consensus.cc:2454] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148107113cf14e118b4170de6bec675a in term 1.
I20260925 15:05:06.791064 31584 raft_consensus.cc:686] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [term 1 LEADER]: Becoming Leader. State: Replica: 148107113cf14e118b4170de6bec675a, State: Running, Role: LEADER
I20260925 15:05:06.791324 31584 consensus_queue.cc:237] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.793730 31066 catalog_manager.cc:5495] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a reported cstate change: term changed from 0 to 1, leader changed from <none> to 148107113cf14e118b4170de6bec675a (127.22.236.1). New cstate: current_term: 1 leader_uuid: "148107113cf14e118b4170de6bec675a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:05:06 dist-test-slave-nsrc krb5kdc[31021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348703, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.62@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
I20260925 15:05:06.871261 31612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:06.858475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37671 (local address 127.0.0.1:59060)
0925 15:05:06.859020 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:06.859035 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:06.859172 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:06.859457 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:06.859462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:06.860102 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:06.860409 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:06.860420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.861378 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.861382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:06.862326 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:06.862338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.862998 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.863004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:06.863141 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.863737 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:06.863754 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:06.866044 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:06.868857 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:06.868868 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:06.868876 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:06.869360 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:06.869666 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:06.869671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:06.869674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:06.869803 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:06.870261 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:06.870268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:06.870426 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 15:05:06.870559 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 15:05:06.870822 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":288,"threads_started":1}
I20260925 15:05:06.871281 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:06.858376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59060 (local address 127.22.236.62:37671)
0925 15:05:06.858570 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:06.858574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:06.859119 (+ 545us) server_negotiation.cc:407] Connection header received
0925 15:05:06.859274 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:06.859279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:06.859338 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:06.859422 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:06.860538 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.861259 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.862446 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:06.862837 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:06.862939 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:06.866230 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:06.866235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:06.866238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:06.866266 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:06.868744 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:06.869463 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:06.869468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:06.869472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:06.869575 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:06.869898 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:06.869903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:06.869907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:06.870144 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:06.870283 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:06.870650 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 15:05:06.870963 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":94}
W20260925 15:05:06.871575 31413 tablet.cc:2369] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:05:06.876578 31066 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:59060:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:05:06.877589 31066 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:05:06.883790 31202 tablet_service.cc:1467] Processing CreateTablet for tablet 68529ef390b04580ba33f8c6fc6db7b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=edaae58dd3384dd38dd286641202999c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:05:06.884084 31341 tablet_service.cc:1467] Processing CreateTablet for tablet 68529ef390b04580ba33f8c6fc6db7b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=edaae58dd3384dd38dd286641202999c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:05:06.884338 31202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68529ef390b04580ba33f8c6fc6db7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:06.884233 31481 tablet_service.cc:1467] Processing CreateTablet for tablet 68529ef390b04580ba33f8c6fc6db7b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=edaae58dd3384dd38dd286641202999c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:05:06.884581 31341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68529ef390b04580ba33f8c6fc6db7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:06.884681 31481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68529ef390b04580ba33f8c6fc6db7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:06.891220 31576 tablet_bootstrap.cc:492] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d: Bootstrap starting.
I20260925 15:05:06.891376 31578 tablet_bootstrap.cc:492] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a: Bootstrap starting.
I20260925 15:05:06.892820 31576 tablet_bootstrap.cc:654] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:06.893115 31578 tablet_bootstrap.cc:654] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:06.894110 31577 tablet_bootstrap.cc:492] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea: Bootstrap starting.
I20260925 15:05:06.894800 31576 tablet_bootstrap.cc:492] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d: No bootstrap required, opened a new log
I20260925 15:05:06.894975 31576 ts_tablet_manager.cc:1397] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
W20260925 15:05:06.895469 31274 tablet.cc:2369] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:05:06.895653 31577 tablet_bootstrap.cc:654] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:06.895747 31576 raft_consensus.cc:348] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.895989 31576 raft_consensus.cc:374] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:06.896067 31576 raft_consensus.cc:729] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1132db8fd64678a3f5847950411e6d, State: Initialized, Role: FOLLOWER
I20260925 15:05:06.896317 31576 consensus_queue.cc:260] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.896723 31583 tablet_replica.cc:439] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } }
I20260925 15:05:06.897022 31583 tablet_replica.cc:442] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:05:06.897900 31577 tablet_bootstrap.cc:492] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea: No bootstrap required, opened a new log
I20260925 15:05:06.898044 31577 ts_tablet_manager.cc:1397] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 15:05:06.898244 31576 ts_tablet_manager.cc:1428] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260925 15:05:06.898698 31577 raft_consensus.cc:348] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.898904 31577 raft_consensus.cc:374] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:06.898977 31577 raft_consensus.cc:729] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7062aeb28dd74e0e9517946ec13a13ea, State: Initialized, Role: FOLLOWER
I20260925 15:05:06.899232 31577 consensus_queue.cc:260] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.899636 31582 tablet_replica.cc:439] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } }
I20260925 15:05:06.899947 31582 tablet_replica.cc:442] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:05:06.900799 31577 ts_tablet_manager.cc:1428] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 15:05:06.908766 31578 tablet_bootstrap.cc:492] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a: No bootstrap required, opened a new log
I20260925 15:05:06.908967 31578 ts_tablet_manager.cc:1397] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.009s
I20260925 15:05:06.909711 31578 raft_consensus.cc:348] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.909976 31578 raft_consensus.cc:374] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:06.910066 31578 raft_consensus.cc:729] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148107113cf14e118b4170de6bec675a, State: Initialized, Role: FOLLOWER
I20260925 15:05:06.910319 31578 consensus_queue.cc:260] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:06.910938 31578 ts_tablet_manager.cc:1428] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 15:05:06.910758 31584 tablet_replica.cc:439] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } }
I20260925 15:05:06.911111 31584 tablet_replica.cc:442] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:05:07.073918 31583 raft_consensus.cc:482] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:05:07.074227 31583 raft_consensus.cc:504] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.075657 31583 leader_election.cc:288] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148107113cf14e118b4170de6bec675a (127.22.236.1:33199), 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833)
I20260925 15:05:07.083138 31574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.076080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:58791 (local address 127.22.236.1:33199)
0925 15:05:07.076279 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:07.076286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:07.077040 (+ 754us) server_negotiation.cc:407] Connection header received
0925 15:05:07.077179 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:07.077183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:07.077242 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:07.077312 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:07.077846 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.078557 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.080987 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.082205 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.082398 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.082466 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:07.082742 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 15:05:07.082880 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:05:07.083415 31617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.075923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:36833 (local address 127.22.236.3:60613)
0925 15:05:07.076635 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:07.076653 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:07.076804 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:07.077471 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:07.077475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:07.077505 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:07.077888 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:07.077900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.078967 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.078973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:07.080929 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:07.080940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.082603 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.082609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:07.082773 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.082802 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:05:07.082896 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:05:07.083016 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":35,"thread_start_us":781,"threads_started":1}
I20260925 15:05:07.083484 31616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.075960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33199 (local address 127.22.236.3:58791)
0925 15:05:07.076948 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:07.076963 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:07.077061 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:07.077349 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:07.077353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:07.077378 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:07.077709 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:07.077717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.078679 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.078684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:07.080831 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:07.080850 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.082383 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.082389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:07.082551 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.082587 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:05:07.082701 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 15:05:07.082836 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":29,"thread_start_us":435,"threads_started":1}
I20260925 15:05:07.083707 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.076069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:60613 (local address 127.22.236.2:36833)
0925 15:05:07.077172 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:07.077176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:07.077192 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:05:07.077246 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:07.077250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:07.077314 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:07.077413 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:07.078044 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.078803 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.081058 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.082252 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.082393 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.082466 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:07.083113 (+ 647us) server_negotiation.cc:299] Negotiation successful
0925 15:05:07.083439 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":923,"threads_started":1}
I20260925 15:05:07.084300 31362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68529ef390b04580ba33f8c6fc6db7b8" candidate_uuid: "5d1132db8fd64678a3f5847950411e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7062aeb28dd74e0e9517946ec13a13ea" is_pre_election: true
I20260925 15:05:07.084405 31227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68529ef390b04580ba33f8c6fc6db7b8" candidate_uuid: "5d1132db8fd64678a3f5847950411e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148107113cf14e118b4170de6bec675a" is_pre_election: true
I20260925 15:05:07.084576 31362 raft_consensus.cc:2454] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1132db8fd64678a3f5847950411e6d in term 0.
I20260925 15:05:07.084757 31227 raft_consensus.cc:2454] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1132db8fd64678a3f5847950411e6d in term 0.
I20260925 15:05:07.085136 31434 leader_election.cc:302] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1132db8fd64678a3f5847950411e6d, 7062aeb28dd74e0e9517946ec13a13ea; no voters:
I20260925 15:05:07.085551 31583 raft_consensus.cc:2781] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:05:07.085672 31583 raft_consensus.cc:482] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:05:07.085752 31583 raft_consensus.cc:3037] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:07.087723 31583 raft_consensus.cc:504] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.088357 31583 leader_election.cc:288] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [CANDIDATE]: Term 1 election: Requested vote from peers 148107113cf14e118b4170de6bec675a (127.22.236.1:33199), 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833)
I20260925 15:05:07.088835 31362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68529ef390b04580ba33f8c6fc6db7b8" candidate_uuid: "5d1132db8fd64678a3f5847950411e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7062aeb28dd74e0e9517946ec13a13ea"
I20260925 15:05:07.089040 31362 raft_consensus.cc:3037] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:07.091653 31362 raft_consensus.cc:2454] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1132db8fd64678a3f5847950411e6d in term 1.
I20260925 15:05:07.092003 31434 leader_election.cc:302] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1132db8fd64678a3f5847950411e6d, 7062aeb28dd74e0e9517946ec13a13ea; no voters:
I20260925 15:05:07.091993 31227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68529ef390b04580ba33f8c6fc6db7b8" candidate_uuid: "5d1132db8fd64678a3f5847950411e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148107113cf14e118b4170de6bec675a"
I20260925 15:05:07.092263 31227 raft_consensus.cc:3037] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:07.092545 31583 raft_consensus.cc:2781] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:05:07.093019 31583 raft_consensus.cc:686] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [term 1 LEADER]: Becoming Leader. State: Replica: 5d1132db8fd64678a3f5847950411e6d, State: Running, Role: LEADER
I20260925 15:05:07.093422 31583 consensus_queue.cc:237] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.095047 31227 raft_consensus.cc:2454] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1132db8fd64678a3f5847950411e6d in term 1.
I20260925 15:05:07.095604 31619 tablet_replica.cc:439] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d: TxnStatusTablet state changed. Reason: New leader 5d1132db8fd64678a3f5847950411e6d. Latest consensus state: current_term: 1 leader_uuid: "5d1132db8fd64678a3f5847950411e6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } }
I20260925 15:05:07.095993 31619 tablet_replica.cc:442] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:05:07.096613 31621 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:05:07.096793 31621 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:05:07.097081 31066 catalog_manager.cc:5495] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d1132db8fd64678a3f5847950411e6d (127.22.236.3). New cstate: current_term: 1 leader_uuid: "5d1132db8fd64678a3f5847950411e6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } health_report { overall_health: HEALTHY } } }
I20260925 15:05:07.142033 31612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.135995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:43511 (local address 127.0.0.1:43030)
0925 15:05:07.136259 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:07.136281 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:07.136411 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:07.137011 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:07.137014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:07.137022 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:05:07.137187 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:07.137194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.138581 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.138585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:07.139531 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:07.139542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.140502 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.140506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:07.140630 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.140680 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:05:07.141673 (+ 993us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:05:07.141684 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:05:07.141729 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 15:05:07.141833 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 15:05:07.142190 31624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.136036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43030 (local address 127.22.236.3:43511)
0925 15:05:07.136635 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:07.136641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:07.136661 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:05:07.136748 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:07.136753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:07.136848 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:07.137009 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:05:07.137272 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.138465 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.139662 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.140375 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.140490 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.140860 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:05:07.141583 (+ 723us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:05:07.141726 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:07.141788 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 15:05:07.141869 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":265,"threads_started":1}
I20260925 15:05:07.146135 31362 raft_consensus.cc:1261] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Refusing update from remote peer 5d1132db8fd64678a3f5847950411e6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:05:07.146294 31227 raft_consensus.cc:1261] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Refusing update from remote peer 5d1132db8fd64678a3f5847950411e6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:05:07.146889 31619 consensus_queue.cc:1035] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:05:07.147204 31583 consensus_queue.cc:1035] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:05:07.153338 31582 tablet_replica.cc:439] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea: TxnStatusTablet state changed. Reason: New leader 5d1132db8fd64678a3f5847950411e6d. Latest consensus state: current_term: 1 leader_uuid: "5d1132db8fd64678a3f5847950411e6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } }
I20260925 15:05:07.153621 31582 tablet_replica.cc:442] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:05:07.167434 31582 tablet_replica.cc:439] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d1132db8fd64678a3f5847950411e6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } }
I20260925 15:05:07.167714 31582 tablet_replica.cc:442] T 68529ef390b04580ba33f8c6fc6db7b8 P 7062aeb28dd74e0e9517946ec13a13ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:05:07.171108 31619 tablet_replica.cc:439] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d1132db8fd64678a3f5847950411e6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } }
I20260925 15:05:07.171387 31619 tablet_replica.cc:442] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:05:07.176906 31584 tablet_replica.cc:439] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a: TxnStatusTablet state changed. Reason: New leader 5d1132db8fd64678a3f5847950411e6d. Latest consensus state: current_term: 1 leader_uuid: "5d1132db8fd64678a3f5847950411e6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } }
I20260925 15:05:07.177176 31584 tablet_replica.cc:442] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:05:07.179474 31619 tablet_replica.cc:439] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d1132db8fd64678a3f5847950411e6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } }
I20260925 15:05:07.179850 31619 tablet_replica.cc:442] T 68529ef390b04580ba33f8c6fc6db7b8 P 5d1132db8fd64678a3f5847950411e6d: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:05:07.186492 31584 tablet_replica.cc:439] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d1132db8fd64678a3f5847950411e6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } } }
I20260925 15:05:07.186739 31584 tablet_replica.cc:442] T 68529ef390b04580ba33f8c6fc6db7b8 P 148107113cf14e118b4170de6bec675a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:05:07.194600 31624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.189556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:59835 (local address 127.22.236.3:43511)
0925 15:05:07.189705 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:07.189711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:07.190154 (+ 443us) server_negotiation.cc:407] Connection header received
0925 15:05:07.190322 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:07.190327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:07.190400 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:07.190470 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:05:07.190915 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.191657 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.193002 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.193441 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.193549 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.193920 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:05:07.194029 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:05:07.194114 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:07.194227 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 15:05:07.194301 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:05:07.194646 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.189566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:43511 (local address 127.22.236.1:59835)
0925 15:05:07.190058 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:07.190073 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:07.190198 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:07.190497 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:07.190502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:07.190514 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:05:07.190774 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:07.190783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.191783 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.191789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:07.192868 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:07.192879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.193577 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.193582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:07.193712 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.193761 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:05:07.194117 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:05:07.194128 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:05:07.194197 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:05:07.194325 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":230,"threads_started":1}
I20260925 15:05:07.203833 31501 raft_consensus.cc:1261] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Refusing update from remote peer 148107113cf14e118b4170de6bec675a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:05:07.203904 31362 raft_consensus.cc:1261] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Refusing update from remote peer 148107113cf14e118b4170de6bec675a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:05:07.204367 31584 consensus_queue.cc:1035] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:05:07.204617 31587 consensus_queue.cc:1035] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:05:07.244169 31574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.237560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:52371 (local address 127.22.236.1:33199)
0925 15:05:07.237728 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:07.237734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:07.238336 (+ 602us) server_negotiation.cc:407] Connection header received
0925 15:05:07.238513 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:07.238519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:07.238593 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:07.238693 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:05:07.239229 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.240250 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.242290 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.242849 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.242968 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.243524 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:05:07.243613 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:05:07.243684 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:07.243812 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 15:05:07.243882 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:05:07.244344 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.237616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33199 (local address 127.22.236.3:52371)
0925 15:05:07.238235 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:07.238254 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:07.238387 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:07.238716 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:07.238721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:07.238733 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:05:07.239070 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:07.239081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.240357 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.240363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:07.242145 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:07.242160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.243020 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.243031 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:07.243186 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.243256 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:05:07.243705 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:05:07.243716 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:05:07.243787 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:05:07.243936 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":296,"threads_started":1}
I20260925 15:05:07.275851 31064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:05:07.276099 31064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:05:07.279222 31064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45: Sending DeleteTablet for 3 replicas of tablet d6d7e36e93f240bbab198931f390ecb2
I20260925 15:05:07.280182 31202 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d7e36e93f240bbab198931f390ecb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:07 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:52542
I20260925 15:05:07.280882 31341 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d7e36e93f240bbab198931f390ecb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:07 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:49350
I20260925 15:05:07.284315 31481 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d7e36e93f240bbab198931f390ecb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:07 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:43670
I20260925 15:05:07.293195 31643 tablet_replica.cc:330] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea: stopping tablet replica
I20260925 15:05:07.293427 31643 raft_consensus.cc:2229] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:05:07.293668 31643 raft_consensus.cc:2258] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:07.294839 31643 ts_tablet_manager.cc:1905] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:07.295758 31644 tablet_replica.cc:330] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d: stopping tablet replica
I20260925 15:05:07.295969 31644 raft_consensus.cc:2229] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:05:07.296255 31644 raft_consensus.cc:2258] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:07.297235 31644 ts_tablet_manager.cc:1905] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:07.300293 31642 tablet_replica.cc:330] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a: stopping tablet replica
I20260925 15:05:07.300486 31642 raft_consensus.cc:2229] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:05:07.300846 31642 raft_consensus.cc:2258] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:07.302266 31643 ts_tablet_manager.cc:1918] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:05:07.302413 31643 log.cc:1196] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/d6d7e36e93f240bbab198931f390ecb2
I20260925 15:05:07.302866 31643 ts_tablet_manager.cc:1939] T d6d7e36e93f240bbab198931f390ecb2 P 7062aeb28dd74e0e9517946ec13a13ea: Deleting consensus metadata
I20260925 15:05:07.303910 31052 catalog_manager.cc:4841] TS 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833): tablet d6d7e36e93f240bbab198931f390ecb2 (table test-table [id=ab0d476a772c426c8828f7533c127bf0]) successfully deleted
I20260925 15:05:07.305142 31644 ts_tablet_manager.cc:1918] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:05:07.305267 31644 log.cc:1196] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/d6d7e36e93f240bbab198931f390ecb2
I20260925 15:05:07.305696 31644 ts_tablet_manager.cc:1939] T d6d7e36e93f240bbab198931f390ecb2 P 5d1132db8fd64678a3f5847950411e6d: Deleting consensus metadata
I20260925 15:05:07.306759 31050 catalog_manager.cc:4841] TS 5d1132db8fd64678a3f5847950411e6d (127.22.236.3:43511): tablet d6d7e36e93f240bbab198931f390ecb2 (table test-table [id=ab0d476a772c426c8828f7533c127bf0]) successfully deleted
I20260925 15:05:07.307375 31642 ts_tablet_manager.cc:1905] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:07.312897 31642 ts_tablet_manager.cc:1918] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:05:07.313009 31642 log.cc:1196] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/d6d7e36e93f240bbab198931f390ecb2
I20260925 15:05:07.313357 31642 ts_tablet_manager.cc:1939] T d6d7e36e93f240bbab198931f390ecb2 P 148107113cf14e118b4170de6bec675a: Deleting consensus metadata
I20260925 15:05:07.314178 31053 catalog_manager.cc:4841] TS 148107113cf14e118b4170de6bec675a (127.22.236.1:33199): tablet d6d7e36e93f240bbab198931f390ecb2 (table test-table [id=ab0d476a772c426c8828f7533c127bf0]) successfully deleted
Sep 25 15:05:07 dist-test-slave-nsrc krb5kdc[31021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:05:07 dist-test-slave-nsrc krb5kdc[31021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:07 dist-test-slave-nsrc krb5kdc[31021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:05:07.345523 31161 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:07.333333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59072 (local address 127.22.236.62:37671)
0925 15:05:07.333553 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:07.333559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:07.333927 (+ 368us) server_negotiation.cc:407] Connection header received
0925 15:05:07.334063 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:07.334067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:07.334128 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:07.334211 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:07.335833 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.336599 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.338382 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.339055 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.339175 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.345257 (+ 6082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59072: BlockingRecv error: recv got EOF from 127.0.0.1:59072 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 15:05:07 dist-test-slave-nsrc krb5kdc[31021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM
I20260925 15:05:07.374219 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.353434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59080 (local address 127.22.236.62:37671)
0925 15:05:07.353603 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:07.353609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:07.357958 (+ 4349us) server_negotiation.cc:407] Connection header received
0925 15:05:07.358137 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:07.358143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:07.358223 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:07.358311 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:07.359807 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.360903 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.362641 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.363256 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.363371 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.368120 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:07.368127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:07.368133 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:07.368217 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:07.371584 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:07.372236 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:07.372242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:07.372246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:07.372406 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:07.372808 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:07.372814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:07.372817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:07.373181 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:07.373437 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:07.373663 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 15:05:07.373873 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:05:07.377758 31064 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:05:07.378400 31064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:05:07.384749 31202 tablet_service.cc:1467] Processing CreateTablet for tablet dd791df0a6fb46278aa5111d77806a3e (DEFAULT_TABLE table=test-table [id=be97ac7badfc492dacd16e098e32ff23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:07.385135 31481 tablet_service.cc:1467] Processing CreateTablet for tablet dd791df0a6fb46278aa5111d77806a3e (DEFAULT_TABLE table=test-table [id=be97ac7badfc492dacd16e098e32ff23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:07.385242 31202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd791df0a6fb46278aa5111d77806a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:07.385229 31341 tablet_service.cc:1467] Processing CreateTablet for tablet dd791df0a6fb46278aa5111d77806a3e (DEFAULT_TABLE table=test-table [id=be97ac7badfc492dacd16e098e32ff23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:07.385540 31481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd791df0a6fb46278aa5111d77806a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:07.385632 31341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd791df0a6fb46278aa5111d77806a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:07.392549 31576 tablet_bootstrap.cc:492] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d: Bootstrap starting.
I20260925 15:05:07.393827 31576 tablet_bootstrap.cc:654] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:07.395565 31576 tablet_bootstrap.cc:492] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d: No bootstrap required, opened a new log
I20260925 15:05:07.395699 31576 ts_tablet_manager.cc:1397] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:05:07.396346 31577 tablet_bootstrap.cc:492] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea: Bootstrap starting.
I20260925 15:05:07.396384 31576 raft_consensus.cc:348] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.396641 31576 raft_consensus.cc:374] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:07.396735 31576 raft_consensus.cc:729] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1132db8fd64678a3f5847950411e6d, State: Initialized, Role: FOLLOWER
I20260925 15:05:07.396957 31576 consensus_queue.cc:260] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.397385 31576 ts_tablet_manager.cc:1428] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:05:07.397869 31577 tablet_bootstrap.cc:654] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:07.401043 31578 tablet_bootstrap.cc:492] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a: Bootstrap starting.
I20260925 15:05:07.402681 31578 tablet_bootstrap.cc:654] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:07.405625 31578 tablet_bootstrap.cc:492] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a: No bootstrap required, opened a new log
I20260925 15:05:07.405774 31578 ts_tablet_manager.cc:1397] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260925 15:05:07.406564 31578 raft_consensus.cc:348] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.406854 31578 raft_consensus.cc:374] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:07.406963 31578 raft_consensus.cc:729] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148107113cf14e118b4170de6bec675a, State: Initialized, Role: FOLLOWER
I20260925 15:05:07.407553 31578 consensus_queue.cc:260] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.408272 31578 ts_tablet_manager.cc:1428] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 15:05:07.413424 31577 tablet_bootstrap.cc:492] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea: No bootstrap required, opened a new log
I20260925 15:05:07.413578 31577 ts_tablet_manager.cc:1397] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20260925 15:05:07.414283 31577 raft_consensus.cc:348] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.414566 31577 raft_consensus.cc:374] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:07.414661 31577 raft_consensus.cc:729] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7062aeb28dd74e0e9517946ec13a13ea, State: Initialized, Role: FOLLOWER
I20260925 15:05:07.414872 31577 consensus_queue.cc:260] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.415392 31577 ts_tablet_manager.cc:1428] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 15:05:07.449589 31587 raft_consensus.cc:482] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:05:07.449757 31587 raft_consensus.cc:504] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.450412 31587 leader_election.cc:288] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833), 5d1132db8fd64678a3f5847950411e6d (127.22.236.3:43511)
I20260925 15:05:07.450903 31362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd791df0a6fb46278aa5111d77806a3e" candidate_uuid: "148107113cf14e118b4170de6bec675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7062aeb28dd74e0e9517946ec13a13ea" is_pre_election: true
I20260925 15:05:07.450932 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd791df0a6fb46278aa5111d77806a3e" candidate_uuid: "148107113cf14e118b4170de6bec675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1132db8fd64678a3f5847950411e6d" is_pre_election: true
I20260925 15:05:07.451146 31362 raft_consensus.cc:2454] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148107113cf14e118b4170de6bec675a in term 0.
I20260925 15:05:07.451174 31501 raft_consensus.cc:2454] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148107113cf14e118b4170de6bec675a in term 0.
I20260925 15:05:07.451608 31153 leader_election.cc:302] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148107113cf14e118b4170de6bec675a, 5d1132db8fd64678a3f5847950411e6d; no voters:
I20260925 15:05:07.451846 31587 raft_consensus.cc:2781] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:05:07.451946 31587 raft_consensus.cc:482] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:05:07.452050 31587 raft_consensus.cc:3037] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:07.453936 31587 raft_consensus.cc:504] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.454408 31587 leader_election.cc:288] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [CANDIDATE]: Term 1 election: Requested vote from peers 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833), 5d1132db8fd64678a3f5847950411e6d (127.22.236.3:43511)
I20260925 15:05:07.454859 31362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd791df0a6fb46278aa5111d77806a3e" candidate_uuid: "148107113cf14e118b4170de6bec675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7062aeb28dd74e0e9517946ec13a13ea"
I20260925 15:05:07.454867 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd791df0a6fb46278aa5111d77806a3e" candidate_uuid: "148107113cf14e118b4170de6bec675a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d1132db8fd64678a3f5847950411e6d"
I20260925 15:05:07.455047 31501 raft_consensus.cc:3037] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:07.455042 31362 raft_consensus.cc:3037] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:07.457844 31362 raft_consensus.cc:2454] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148107113cf14e118b4170de6bec675a in term 1.
I20260925 15:05:07.457943 31501 raft_consensus.cc:2454] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148107113cf14e118b4170de6bec675a in term 1.
I20260925 15:05:07.458253 31155 leader_election.cc:302] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148107113cf14e118b4170de6bec675a, 7062aeb28dd74e0e9517946ec13a13ea; no voters:
I20260925 15:05:07.458559 31587 raft_consensus.cc:2781] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:05:07.458773 31587 raft_consensus.cc:686] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [term 1 LEADER]: Becoming Leader. State: Replica: 148107113cf14e118b4170de6bec675a, State: Running, Role: LEADER
I20260925 15:05:07.459028 31587 consensus_queue.cc:237] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.461534 31064 catalog_manager.cc:5495] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a reported cstate change: term changed from 0 to 1, leader changed from <none> to 148107113cf14e118b4170de6bec675a (127.22.236.1). New cstate: current_term: 1 leader_uuid: "148107113cf14e118b4170de6bec675a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } health_report { overall_health: UNKNOWN } } }
I20260925 15:05:07.480786 31574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.476234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.22.236.1:33199)
0925 15:05:07.476412 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:07.476419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:07.476439 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:05:07.476589 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:07.476594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:07.476677 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:07.476777 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:05:07.477101 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.477883 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.479212 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.479623 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.479725 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.480098 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:05:07.480254 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:05:07.480327 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:07.480456 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 15:05:07.480528 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:05:07.483319 31362 raft_consensus.cc:1261] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Refusing update from remote peer 148107113cf14e118b4170de6bec675a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:05:07.483320 31501 raft_consensus.cc:1261] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Refusing update from remote peer 148107113cf14e118b4170de6bec675a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:05:07.483842 31584 consensus_queue.cc:1035] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:05:07.484086 31587 consensus_queue.cc:1035] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:05:07.499778 31066 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:05:07.500042 31066 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:05:07.503015 31066 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45: Sending DeleteTablet for 3 replicas of tablet dd791df0a6fb46278aa5111d77806a3e
I20260925 15:05:07.503675 31202 tablet_service.cc:1514] Processing DeleteTablet for tablet dd791df0a6fb46278aa5111d77806a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:07 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:52542
I20260925 15:05:07.503943 31642 tablet_replica.cc:330] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a: stopping tablet replica
I20260925 15:05:07.503958 31341 tablet_service.cc:1514] Processing DeleteTablet for tablet dd791df0a6fb46278aa5111d77806a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:07 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:49350
I20260925 15:05:07.504155 31642 raft_consensus.cc:2229] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:05:07.504289 31481 tablet_service.cc:1514] Processing DeleteTablet for tablet dd791df0a6fb46278aa5111d77806a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:07 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:43670
I20260925 15:05:07.504468 31642 raft_consensus.cc:2258] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:07.504557 31644 tablet_replica.cc:330] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d: stopping tablet replica
I20260925 15:05:07.504726 31643 tablet_replica.cc:330] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea: stopping tablet replica
I20260925 15:05:07.504750 31644 raft_consensus.cc:2229] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:05:07.504921 31643 raft_consensus.cc:2229] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:05:07.504981 31644 raft_consensus.cc:2258] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:07.505118 31643 raft_consensus.cc:2258] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:07.506209 31643 ts_tablet_manager.cc:1905] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:07.506209 31642 ts_tablet_manager.cc:1905] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:07.506175 31066 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:05:07.506769 31066 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:05:07.512493 31642 ts_tablet_manager.cc:1918] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:05:07.512643 31642 log.cc:1196] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/dd791df0a6fb46278aa5111d77806a3e
I20260925 15:05:07.513019 31642 ts_tablet_manager.cc:1939] T dd791df0a6fb46278aa5111d77806a3e P 148107113cf14e118b4170de6bec675a: Deleting consensus metadata
I20260925 15:05:07.513089 31643 ts_tablet_manager.cc:1918] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:05:07.513206 31643 log.cc:1196] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/dd791df0a6fb46278aa5111d77806a3e
I20260925 15:05:07.513563 31643 ts_tablet_manager.cc:1939] T dd791df0a6fb46278aa5111d77806a3e P 7062aeb28dd74e0e9517946ec13a13ea: Deleting consensus metadata
I20260925 15:05:07.514086 31053 catalog_manager.cc:4841] TS 148107113cf14e118b4170de6bec675a (127.22.236.1:33199): tablet dd791df0a6fb46278aa5111d77806a3e (table test-table [id=be97ac7badfc492dacd16e098e32ff23]) successfully deleted
I20260925 15:05:07.514488 31052 catalog_manager.cc:4841] TS 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833): tablet dd791df0a6fb46278aa5111d77806a3e (table test-table [id=be97ac7badfc492dacd16e098e32ff23]) successfully deleted
I20260925 15:05:07.515393 31644 ts_tablet_manager.cc:1905] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:07.517922 31202 tablet_service.cc:1467] Processing CreateTablet for tablet 2b01e70bde9f4b51bb906d4de7ebb884 (DEFAULT_TABLE table=test-table [id=c57c38f268c34883aba43c3a52ab8f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:07.518276 31202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b01e70bde9f4b51bb906d4de7ebb884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:07.518335 31481 tablet_service.cc:1467] Processing CreateTablet for tablet 2b01e70bde9f4b51bb906d4de7ebb884 (DEFAULT_TABLE table=test-table [id=c57c38f268c34883aba43c3a52ab8f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:07.518362 31341 tablet_service.cc:1467] Processing CreateTablet for tablet 2b01e70bde9f4b51bb906d4de7ebb884 (DEFAULT_TABLE table=test-table [id=c57c38f268c34883aba43c3a52ab8f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:07.518704 31341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b01e70bde9f4b51bb906d4de7ebb884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:07.518750 31481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b01e70bde9f4b51bb906d4de7ebb884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:07.524786 31577 tablet_bootstrap.cc:492] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea: Bootstrap starting.
I20260925 15:05:07.526264 31577 tablet_bootstrap.cc:654] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:07.526473 31576 tablet_bootstrap.cc:492] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d: Bootstrap starting.
I20260925 15:05:07.528126 31576 tablet_bootstrap.cc:654] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:07.529853 31644 ts_tablet_manager.cc:1918] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:05:07.530005 31644 log.cc:1196] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/dd791df0a6fb46278aa5111d77806a3e
I20260925 15:05:07.530416 31576 tablet_bootstrap.cc:492] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d: No bootstrap required, opened a new log
I20260925 15:05:07.530540 31644 ts_tablet_manager.cc:1939] T dd791df0a6fb46278aa5111d77806a3e P 5d1132db8fd64678a3f5847950411e6d: Deleting consensus metadata
I20260925 15:05:07.530561 31576 ts_tablet_manager.cc:1397] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 15:05:07.531380 31050 catalog_manager.cc:4841] TS 5d1132db8fd64678a3f5847950411e6d (127.22.236.3:43511): tablet dd791df0a6fb46278aa5111d77806a3e (table test-table [id=be97ac7badfc492dacd16e098e32ff23]) successfully deleted
I20260925 15:05:07.531303 31576 raft_consensus.cc:348] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.531548 31576 raft_consensus.cc:374] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:07.531618 31576 raft_consensus.cc:729] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1132db8fd64678a3f5847950411e6d, State: Initialized, Role: FOLLOWER
I20260925 15:05:07.531831 31576 consensus_queue.cc:260] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.532325 31576 ts_tablet_manager.cc:1428] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:05:07.533463 31578 tablet_bootstrap.cc:492] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a: Bootstrap starting.
I20260925 15:05:07.534979 31578 tablet_bootstrap.cc:654] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:07.536888 31578 tablet_bootstrap.cc:492] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a: No bootstrap required, opened a new log
I20260925 15:05:07.537052 31578 ts_tablet_manager.cc:1397] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260925 15:05:07.537802 31578 raft_consensus.cc:348] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.538069 31578 raft_consensus.cc:374] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:07.538134 31578 raft_consensus.cc:729] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148107113cf14e118b4170de6bec675a, State: Initialized, Role: FOLLOWER
I20260925 15:05:07.538530 31578 consensus_queue.cc:260] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.539080 31578 ts_tablet_manager.cc:1428] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 15:05:07.541426 31577 tablet_bootstrap.cc:492] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea: No bootstrap required, opened a new log
I20260925 15:05:07.541602 31577 ts_tablet_manager.cc:1397] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.006s
I20260925 15:05:07.542377 31577 raft_consensus.cc:348] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.542624 31577 raft_consensus.cc:374] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:07.542702 31577 raft_consensus.cc:729] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7062aeb28dd74e0e9517946ec13a13ea, State: Initialized, Role: FOLLOWER
I20260925 15:05:07.542937 31577 consensus_queue.cc:260] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.543545 31577 ts_tablet_manager.cc:1428] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 15:05:07.644997 31633 raft_consensus.cc:482] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:05:07.645205 31633 raft_consensus.cc:504] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.645841 31633 leader_election.cc:288] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148107113cf14e118b4170de6bec675a (127.22.236.1:33199), 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833)
I20260925 15:05:07.646347 31227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b01e70bde9f4b51bb906d4de7ebb884" candidate_uuid: "5d1132db8fd64678a3f5847950411e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148107113cf14e118b4170de6bec675a" is_pre_election: true
I20260925 15:05:07.646412 31362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b01e70bde9f4b51bb906d4de7ebb884" candidate_uuid: "5d1132db8fd64678a3f5847950411e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7062aeb28dd74e0e9517946ec13a13ea" is_pre_election: true
I20260925 15:05:07.646626 31362 raft_consensus.cc:2454] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1132db8fd64678a3f5847950411e6d in term 0.
I20260925 15:05:07.646632 31227 raft_consensus.cc:2454] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1132db8fd64678a3f5847950411e6d in term 0.
I20260925 15:05:07.647078 31434 leader_election.cc:302] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1132db8fd64678a3f5847950411e6d, 7062aeb28dd74e0e9517946ec13a13ea; no voters:
I20260925 15:05:07.647334 31633 raft_consensus.cc:2781] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:05:07.647455 31633 raft_consensus.cc:482] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:05:07.647537 31633 raft_consensus.cc:3037] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:07.649413 31633 raft_consensus.cc:504] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.649860 31633 leader_election.cc:288] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [CANDIDATE]: Term 1 election: Requested vote from peers 148107113cf14e118b4170de6bec675a (127.22.236.1:33199), 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833)
I20260925 15:05:07.650295 31227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b01e70bde9f4b51bb906d4de7ebb884" candidate_uuid: "5d1132db8fd64678a3f5847950411e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148107113cf14e118b4170de6bec675a"
I20260925 15:05:07.650296 31362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b01e70bde9f4b51bb906d4de7ebb884" candidate_uuid: "5d1132db8fd64678a3f5847950411e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7062aeb28dd74e0e9517946ec13a13ea"
I20260925 15:05:07.650497 31362 raft_consensus.cc:3037] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:07.650497 31227 raft_consensus.cc:3037] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:07.653424 31362 raft_consensus.cc:2454] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1132db8fd64678a3f5847950411e6d in term 1.
I20260925 15:05:07.653419 31227 raft_consensus.cc:2454] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1132db8fd64678a3f5847950411e6d in term 1.
I20260925 15:05:07.653769 31434 leader_election.cc:302] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1132db8fd64678a3f5847950411e6d, 7062aeb28dd74e0e9517946ec13a13ea; no voters:
I20260925 15:05:07.654078 31633 raft_consensus.cc:2781] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:05:07.654228 31633 raft_consensus.cc:686] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [term 1 LEADER]: Becoming Leader. State: Replica: 5d1132db8fd64678a3f5847950411e6d, State: Running, Role: LEADER
I20260925 15:05:07.654489 31633 consensus_queue.cc:237] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } }
I20260925 15:05:07.656690 31066 catalog_manager.cc:5495] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d1132db8fd64678a3f5847950411e6d (127.22.236.3). New cstate: current_term: 1 leader_uuid: "5d1132db8fd64678a3f5847950411e6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d1132db8fd64678a3f5847950411e6d" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 43511 } health_report { overall_health: HEALTHY } } }
I20260925 15:05:07.692086 31624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.688010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.22.236.3:43511)
0925 15:05:07.688220 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:07.688226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:07.688377 (+ 151us) server_negotiation.cc:407] Connection header received
0925 15:05:07.688502 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:07.688505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:07.688571 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:07.688639 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:05:07.688901 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.689583 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.690763 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.691134 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.691218 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.691539 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:05:07.691631 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:05:07.691689 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:07.691768 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 15:05:07.691822 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:05:07.702535 31624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.698577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:50919 (local address 127.22.236.3:43511)
0925 15:05:07.698741 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:07.698747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:07.698802 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:05:07.698938 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:07.698942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:07.699011 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:07.699095 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:05:07.699350 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.700000 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.701196 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.701661 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.701748 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.702038 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:05:07.702120 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:05:07.702177 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:07.702264 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 15:05:07.702317 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":41}
I20260925 15:05:07.702553 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.698592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:43511 (local address 127.22.236.3:50919)
0925 15:05:07.698739 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:07.698753 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:07.698837 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:07.699098 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:07.699101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:07.699108 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:05:07.699254 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:07.699260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.700100 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.700104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:07.701078 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:07.701089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.701774 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.701777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:07.701890 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.701921 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:05:07.702190 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:05:07.702197 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:05:07.702247 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 15:05:07.702337 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":14}
I20260925 15:05:07.709292 31227 raft_consensus.cc:1261] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Refusing update from remote peer 5d1132db8fd64678a3f5847950411e6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:05:07.709355 31362 raft_consensus.cc:1261] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Refusing update from remote peer 5d1132db8fd64678a3f5847950411e6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:05:07.709815 31583 consensus_queue.cc:1035] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "148107113cf14e118b4170de6bec675a" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:05:07.710045 31619 consensus_queue.cc:1035] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7062aeb28dd74e0e9517946ec13a13ea" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:05:07.752866 31065 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:05:07.753118 31065 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:05:07.755957 31065 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0af0892f9c264ab69bb60e77a49d3d45: Sending DeleteTablet for 3 replicas of tablet 2b01e70bde9f4b51bb906d4de7ebb884
I20260925 15:05:07.756704 31341 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b01e70bde9f4b51bb906d4de7ebb884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:07 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:49350
I20260925 15:05:07.756739 31202 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b01e70bde9f4b51bb906d4de7ebb884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:07 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:52542
I20260925 15:05:07.756996 31643 tablet_replica.cc:330] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea: stopping tablet replica
I20260925 15:05:07.757001 31642 tablet_replica.cc:330] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a: stopping tablet replica
I20260925 15:05:07.757164 31642 raft_consensus.cc:2229] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:05:07.757164 31643 raft_consensus.cc:2229] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:05:07.757344 31643 raft_consensus.cc:2258] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:07.757350 31642 raft_consensus.cc:2258] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:07.757763 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 31137
I20260925 15:05:07.758189 31480 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b01e70bde9f4b51bb906d4de7ebb884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:07 UTC) from {username='oryx', principal='oryx/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:43670
I20260925 15:05:07.758499 31644 tablet_replica.cc:330] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d: stopping tablet replica
I20260925 15:05:07.758678 31644 raft_consensus.cc:2229] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:05:07.759155 31644 raft_consensus.cc:2258] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:07.759239 31642 ts_tablet_manager.cc:1905] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:07.765632 31644 ts_tablet_manager.cc:1905] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:07.765615 31643 ts_tablet_manager.cc:1905] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:07.771904 31643 ts_tablet_manager.cc:1918] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:05:07.772037 31643 log.cc:1196] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/2b01e70bde9f4b51bb906d4de7ebb884
I20260925 15:05:07.772215 31644 ts_tablet_manager.cc:1918] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:05:07.772372 31644 log.cc:1196] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/2b01e70bde9f4b51bb906d4de7ebb884
I20260925 15:05:07.772459 31643 ts_tablet_manager.cc:1939] T 2b01e70bde9f4b51bb906d4de7ebb884 P 7062aeb28dd74e0e9517946ec13a13ea: Deleting consensus metadata
I20260925 15:05:07.772964 31644 ts_tablet_manager.cc:1939] T 2b01e70bde9f4b51bb906d4de7ebb884 P 5d1132db8fd64678a3f5847950411e6d: Deleting consensus metadata
I20260925 15:05:07.773458 31052 catalog_manager.cc:4841] TS 7062aeb28dd74e0e9517946ec13a13ea (127.22.236.2:36833): tablet 2b01e70bde9f4b51bb906d4de7ebb884 (table test-table [id=c57c38f268c34883aba43c3a52ab8f1f]) successfully deleted
I20260925 15:05:07.774384 31050 catalog_manager.cc:4841] TS 5d1132db8fd64678a3f5847950411e6d (127.22.236.3:43511): tablet 2b01e70bde9f4b51bb906d4de7ebb884 (table test-table [id=c57c38f268c34883aba43c3a52ab8f1f]) successfully deleted
I20260925 15:05:07.774665 31642 ts_tablet_manager.cc:1918] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:05:07.774776 31642 log.cc:1196] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/2b01e70bde9f4b51bb906d4de7ebb884
I20260925 15:05:07.775090 31642 ts_tablet_manager.cc:1939] T 2b01e70bde9f4b51bb906d4de7ebb884 P 148107113cf14e118b4170de6bec675a: Deleting consensus metadata
Sep 25 15:05:07 dist-test-slave-nsrc krb5kdc[31021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.236.1@KRBTEST.COM
I20260925 15:05:07.775992 31053 catalog_manager.cc:4841] TS 148107113cf14e118b4170de6bec675a (127.22.236.1:33199): tablet 2b01e70bde9f4b51bb906d4de7ebb884 (table test-table [id=c57c38f268c34883aba43c3a52ab8f1f]) successfully deleted
I20260925 15:05:07.782236 31574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.759986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45066 (local address 127.22.236.1:33199)
0925 15:05:07.760190 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:07.760196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:07.765663 (+ 5467us) server_negotiation.cc:407] Connection header received
0925 15:05:07.765964 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:07.765969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:07.766057 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:07.766139 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:07.767842 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.768967 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.770784 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.771382 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.771503 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.776917 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:07.776931 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:07.776968 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:07.777006 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:07.779749 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:07.780491 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:07.780498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:07.780500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:07.780622 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:07.780975 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:07.780977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:07.780979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:07.781276 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:07.781478 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:07.781804 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 15:05:07.781976 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:05:07.782876 31268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:07.901872 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 31137
I20260925 15:05:07.920560 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 31277
Sep 25 15:05:07 dist-test-slave-nsrc krb5kdc[31021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.236.2@KRBTEST.COM
I20260925 15:05:07.936616 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:07.921287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55680 (local address 127.22.236.2:36833)
0925 15:05:07.921755 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:07.921761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:07.921800 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:05:07.921861 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:07.921867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:07.922016 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:07.922127 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:07.923432 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.924593 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.925784 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:07.926463 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:07.926591 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:07.930481 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:07.930498 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:07.930518 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:07.930562 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:07.934112 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:07.934680 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:07.934690 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:07.934694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:07.934845 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:07.935144 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:07.935149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:07.935153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:07.935563 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:07.935843 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:07.936005 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 15:05:07.936235 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":211,"threads_started":1}
I20260925 15:05:07.937256 31407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:08.029150 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 31277
I20260925 15:05:08.043260 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 31416
Sep 25 15:05:08 dist-test-slave-nsrc krb5kdc[31021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.236.3@KRBTEST.COM
I20260925 15:05:08.060616 31624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:08.043942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.22.236.3:43511)
0925 15:05:08.044162 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:08.044169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:08.044224 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:05:08.044407 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:08.044410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:08.044471 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:08.044559 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:08.045737 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:08.046532 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:08.048450 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:08.048888 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:08.048976 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:08.053904 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:08.053924 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:08.053946 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:08.054007 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:08.057853 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:08.058473 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:08.058482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:08.058486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:08.058668 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:08.058995 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:08.059001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:08.059004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:08.059455 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:08.059731 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:08.059893 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 15:05:08.060103 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:05:08.061287 31546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:08.206583 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 31416
I20260925 15:05:08.206640 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:08.205801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33199 (local address 127.22.236.3:42641)
0925 15:05:08.206268 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:08.206439 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":267,"threads_started":1}
I20260925 15:05:08.225238 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 31034
I20260925 15:05:08.226271 31106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:08.336478 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 31034
2026-09-25T15:05:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6116 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:05:08 dist-test-slave-nsrc krb5kdc[31682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:05:08 dist-test-slave-nsrc krb5kdc[31682](info): set up 2 sockets
Sep 25 15:05:08 dist-test-slave-nsrc krb5kdc[31682](info): commencing operation
krb5kdc: starting...
W20260925 15:05:09.340821 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:05:09 dist-test-slave-nsrc krb5kdc[31682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:05:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:05:09.531950 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:41659
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:41659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:05:09.842916 31695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:09.843199 31695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:09.843288 31695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:09.850685 31695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:05:09.850796 31695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:09.850854 31695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:09.850909 31695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:05:09.850955 31695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:05:09.860611 31695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:41659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:41659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:09.861788 31695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:09.863003 31695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:09.874042 31700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:09.874001 31701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:09.874010 31703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:09.874841 31695 server_base.cc:1034] running on GCE node
I20260925 15:05:09.875581 31695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:09.876911 31695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:09.878105 31695 hybrid_clock.cc:648] HybridClock initialized: now 1790348709878061 us; error 34 us; skew 500 ppm
I20260925 15:05:09.878427 31695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T15:05:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1605 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:05:10 dist-test-slave-nsrc krb5kdc[31709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:05:10 dist-test-slave-nsrc krb5kdc[31709](info): set up 2 sockets
Sep 25 15:05:10 dist-test-slave-nsrc krb5kdc[31709](info): commencing operation
krb5kdc: starting...
W20260925 15:05:10.940864 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:05:11 dist-test-slave-nsrc krb5kdc[31709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:05:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:05:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:05:11.135779 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:38839
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:38839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:05:11.440501 31722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:11.440781 31722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:11.440874 31722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:11.447957 31722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:05:11.448064 31722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:11.448114 31722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:11.448187 31722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:05:11.448230 31722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:05:11.457712 31722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:38839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:38839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:11.458890 31722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:11.460119 31722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:11.470886 31728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:11.470911 31727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:11.471362 31730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:11.471632 31722 server_base.cc:1034] running on GCE node
I20260925 15:05:11.472400 31722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:11.473834 31722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:11.475037 31722 hybrid_clock.cc:648] HybridClock initialized: now 1790348711474986 us; error 42 us; skew 500 ppm
I20260925 15:05:11.475342 31722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:11 dist-test-slave-nsrc krb5kdc[31709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348711, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:11.481889 31722 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:05:11.483292 31722 webserver.cc:460] Webserver started at http://127.22.236.62:34743/ using document root <none> and password file <none>
I20260925 15:05:11.483937 31722 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:11.484046 31722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:11.484350 31722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:11.487146 31722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "ca5d787fb42d435694f0aa0e951ab2d3"
format_stamp: "Formatted at 2026-09-25 15:05:11 on dist-test-slave-nsrc"
server_key: "5c4b7a861eeacaf9c94371ab7f7d6b77"
server_key_iv: "a40e75e67a8c95e64e84b6fc27c6f15b"
server_key_version: "encryptionkey@0"
I20260925 15:05:11.487846 31722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "ca5d787fb42d435694f0aa0e951ab2d3"
format_stamp: "Formatted at 2026-09-25 15:05:11 on dist-test-slave-nsrc"
server_key: "5c4b7a861eeacaf9c94371ab7f7d6b77"
server_key_iv: "a40e75e67a8c95e64e84b6fc27c6f15b"
server_key_version: "encryptionkey@0"
I20260925 15:05:11.492707 31722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:05:11.495936 31737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:11.497193 31722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:05:11.497345 31722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "ca5d787fb42d435694f0aa0e951ab2d3"
format_stamp: "Formatted at 2026-09-25 15:05:11 on dist-test-slave-nsrc"
server_key: "5c4b7a861eeacaf9c94371ab7f7d6b77"
server_key_iv: "a40e75e67a8c95e64e84b6fc27c6f15b"
server_key_version: "encryptionkey@0"
I20260925 15:05:11.497534 31722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:11.518769 31722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:11.522416 31722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:11.522670 31722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:11.538864 31722 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:38839
I20260925 15:05:11.538877 31788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:38839 every 8 connection(s)
I20260925 15:05:11.540156 31722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:05:11.542249 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 31722
I20260925 15:05:11.542446 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:05:11.543062 23472 external_mini_cluster.cc:1428] Setting key 766150ac34c0e0d3e3695b815557415d
I20260925 15:05:11.543718 31789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:11.555559 31789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3: Bootstrap starting.
Sep 25 15:05:11 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:05:11.558791 31789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:11.559768 31789 log.cc:824] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:05:11.562361 31789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3: No bootstrap required, opened a new log
I20260925 15:05:11.563313 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:11.544937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.22.236.62:38839)
0925 15:05:11.545649 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:11.545661 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:11.545701 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:05:11.547184 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:11.547204 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:11.547594 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:11.547975 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:11.549364 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:11.550664 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:11.551872 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:11.552618 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:11.552791 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:11.556606 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:11.556614 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:11.556629 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:11.556668 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:11.559518 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:11.560204 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:11.560210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:11.560217 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:11.560389 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:11.560752 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:11.560756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:11.560758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:11.561249 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:11.561505 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:11.561954 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 15:05:11.562335 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":306,"threads_started":1}
I20260925 15:05:11.569754 31789 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38839 } }
I20260925 15:05:11.570130 31789 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:11.570238 31789 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5d787fb42d435694f0aa0e951ab2d3, State: Initialized, Role: FOLLOWER
I20260925 15:05:11.570803 31789 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38839 } }
I20260925 15:05:11.571003 31789 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:05:11.571108 31789 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:05:11.571230 31789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:11.573503 31789 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38839 } }
I20260925 15:05:11.573974 31789 leader_election.cc:302] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca5d787fb42d435694f0aa0e951ab2d3; no voters:
I20260925 15:05:11.574517 31789 leader_election.cc:288] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:05:11.574651 31794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:05:11.575670 31794 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 1 LEADER]: Becoming Leader. State: Replica: ca5d787fb42d435694f0aa0e951ab2d3, State: Running, Role: LEADER
I20260925 15:05:11.576198 31794 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38839 } }
I20260925 15:05:11.576709 31789 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:05:11.578747 31795 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38839 } } }
I20260925 15:05:11.579137 31795 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:05:11.578992 31796 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca5d787fb42d435694f0aa0e951ab2d3. Latest consensus state: current_term: 1 leader_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38839 } } }
I20260925 15:05:11.579322 31796 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:05:11.580303 31802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:05:11.585583 31802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:05:11.594915 31802 catalog_manager.cc:1347] Generated new cluster ID: 8d5ee3a0ea764aed859865afede59f49
I20260925 15:05:11.595034 31802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:05:11.611413 31802 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:05:11.612725 31802 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:05:11.617720 31802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3: Generated new TSK 0
I20260925 15:05:11.618552 31802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:05:11.675431 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:38839
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:05:12.007300 31815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:12.007567 31815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:12.007658 31815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:12.014793 31815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:12.014900 31815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:12.015021 31815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:05:12.024478 31815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:38839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:12.025722 31815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:12.026968 31815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:12.038973 31820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:12.038997 31821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:12.038925 31823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:12.039813 31815 server_base.cc:1034] running on GCE node
I20260925 15:05:12.040381 31815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:12.041184 31815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:12.042361 31815 hybrid_clock.cc:648] HybridClock initialized: now 1790348712042317 us; error 38 us; skew 500 ppm
I20260925 15:05:12.042635 31815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:12 dist-test-slave-nsrc krb5kdc[31709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348712, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:12.045521 31815 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:05:12.046847 31815 webserver.cc:460] Webserver started at http://127.22.236.1:35853/ using document root <none> and password file <none>
I20260925 15:05:12.047494 31815 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:12.047603 31815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:12.047873 31815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:12.050607 31815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "f11e77284cbf47fcbae02797d55773f1"
format_stamp: "Formatted at 2026-09-25 15:05:12 on dist-test-slave-nsrc"
server_key: "23e5fa385b87ab7aa330962254933eba"
server_key_iv: "b6ba0646955defa87b54834597eddfdf"
server_key_version: "encryptionkey@0"
I20260925 15:05:12.051333 31815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "f11e77284cbf47fcbae02797d55773f1"
format_stamp: "Formatted at 2026-09-25 15:05:12 on dist-test-slave-nsrc"
server_key: "23e5fa385b87ab7aa330962254933eba"
server_key_iv: "b6ba0646955defa87b54834597eddfdf"
server_key_version: "encryptionkey@0"
I20260925 15:05:12.056193 31815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 15:05:12.059397 31830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:12.060995 31815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:05:12.061149 31815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "f11e77284cbf47fcbae02797d55773f1"
format_stamp: "Formatted at 2026-09-25 15:05:12 on dist-test-slave-nsrc"
server_key: "23e5fa385b87ab7aa330962254933eba"
server_key_iv: "b6ba0646955defa87b54834597eddfdf"
server_key_version: "encryptionkey@0"
I20260925 15:05:12.061308 31815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:12.074996 31815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:12.079042 31815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:12.079262 31815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:12.080202 31815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:12.081442 31815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:12.081511 31815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:12.081598 31815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:12.081641 31815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:12.110653 31815 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:38401
I20260925 15:05:12.110697 31942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:38401 every 8 connection(s)
I20260925 15:05:12.111871 31815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:05:12.118520 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 31815
I20260925 15:05:12.118710 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:05:12.119439 23472 external_mini_cluster.cc:1428] Setting key 09cfd01271ad8150891abc087eb91490
Sep 25 15:05:12 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348712, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:05:12.131421 31945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:12.114276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:56437 (local address 127.22.236.62:38839)
0925 15:05:12.114658 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:12.114662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:12.115519 (+ 857us) server_negotiation.cc:407] Connection header received
0925 15:05:12.117298 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:12.117313 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:12.117393 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:12.117489 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:12.119223 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:12.119944 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:12.121693 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:12.122253 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:12.122379 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:12.125647 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:12.125654 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:12.125658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:12.125695 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:12.128410 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:12.129228 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:12.129231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:12.129234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:12.129341 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:12.129841 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:12.129843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:12.129845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:12.130097 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:12.130200 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:12.131015 (+ 815us) server_negotiation.cc:299] Negotiation successful
0925 15:05:12.131185 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":198,"threads_started":1}
I20260925 15:05:12.132792 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:12.114701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:56437)
0925 15:05:12.115363 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:12.115399 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:12.117040 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:12.117711 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:12.117719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:12.118265 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:12.119047 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:12.119072 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:12.120091 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:12.120097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:12.121556 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:12.121567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:12.122472 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:12.122477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:12.122683 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:12.123355 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:12.123378 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:12.125436 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:12.128540 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:12.128550 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:12.128567 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:12.129123 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:12.129498 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:12.129503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:12.129508 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:12.129743 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:12.130207 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:12.130214 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:12.130626 (+ 412us) client_negotiation.cc:769] Sending connection context
0925 15:05:12.130908 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 15:05:12.131436 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":253,"threads_started":1}
I20260925 15:05:12.134253 31943 heartbeater.cc:347] Connected to a master server at 127.22.236.62:38839
I20260925 15:05:12.134542 31943 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:12.135222 31943 heartbeater.cc:511] Master 127.22.236.62:38839 requested a full tablet report, sending...
I20260925 15:05:12.137154 31754 ts_manager.cc:194] Registered new tserver with Master: f11e77284cbf47fcbae02797d55773f1 (127.22.236.1:38401)
I20260925 15:05:12.138517 31754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:56437
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:05:12.165297 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:38839
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:05:12.481817 31949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:12.482095 31949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:12.482226 31949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:12.489333 31949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:12.489449 31949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:12.489606 31949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:05:12.499246 31949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:38839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:12.500546 31949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:12.501894 31949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:12.513974 31954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:12.514009 31955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:12.514050 31957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:12.514819 31949 server_base.cc:1034] running on GCE node
I20260925 15:05:12.515352 31949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:12.516264 31949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:12.517467 31949 hybrid_clock.cc:648] HybridClock initialized: now 1790348712517418 us; error 42 us; skew 500 ppm
I20260925 15:05:12.517751 31949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:12 dist-test-slave-nsrc krb5kdc[31709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348712, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:12.520565 31949 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:05:12.521894 31949 webserver.cc:460] Webserver started at http://127.22.236.2:42627/ using document root <none> and password file <none>
I20260925 15:05:12.522535 31949 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:12.522646 31949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:12.522926 31949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:12.525687 31949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "31222a7e342841d4a00a519fb73625b5"
format_stamp: "Formatted at 2026-09-25 15:05:12 on dist-test-slave-nsrc"
server_key: "ecb6a14ec67d4fdcba8d626e1127df96"
server_key_iv: "99023a8810cba4217b5a49e6124687af"
server_key_version: "encryptionkey@0"
I20260925 15:05:12.526453 31949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "31222a7e342841d4a00a519fb73625b5"
format_stamp: "Formatted at 2026-09-25 15:05:12 on dist-test-slave-nsrc"
server_key: "ecb6a14ec67d4fdcba8d626e1127df96"
server_key_iv: "99023a8810cba4217b5a49e6124687af"
server_key_version: "encryptionkey@0"
I20260925 15:05:12.531183 31949 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:05:12.534343 31964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:12.535573 31949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:05:12.535692 31949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "31222a7e342841d4a00a519fb73625b5"
format_stamp: "Formatted at 2026-09-25 15:05:12 on dist-test-slave-nsrc"
server_key: "ecb6a14ec67d4fdcba8d626e1127df96"
server_key_iv: "99023a8810cba4217b5a49e6124687af"
server_key_version: "encryptionkey@0"
I20260925 15:05:12.535842 31949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:12.555033 31949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:12.558559 31949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:12.558830 31949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:12.559703 31949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:12.561107 31949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:12.561175 31949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:12.561293 31949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:12.561340 31949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:12.590323 31949 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:34215
I20260925 15:05:12.590358 32076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:34215 every 8 connection(s)
I20260925 15:05:12.591528 31949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:05:12.592820 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 31949
I20260925 15:05:12.592947 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:05:12.593329 23472 external_mini_cluster.cc:1428] Setting key c69c8b64ec5765f690a748443b0df5bc
Sep 25 15:05:12 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348712, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
I20260925 15:05:12.611074 31945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:12.595301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:50501 (local address 127.22.236.62:38839)
0925 15:05:12.595453 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:12.595457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:12.596868 (+ 1411us) server_negotiation.cc:407] Connection header received
0925 15:05:12.598576 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:12.598581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:12.598658 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:12.598739 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:12.600127 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:12.600906 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:12.602426 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:12.602830 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:12.602930 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:12.606094 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:12.606099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:12.606102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:12.606133 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:12.608409 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:12.609034 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:12.609039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:12.609043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:12.609160 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:12.609520 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:12.609525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:12.609528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:12.609772 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:12.609904 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:12.610521 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 15:05:12.610740 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:05:12.612066 32079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:12.595936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:50501)
0925 15:05:12.596733 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:12.596769 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:12.598333 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:12.598967 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:12.598975 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:12.599446 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:12.599977 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:12.599993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:12.601032 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:12.601036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:12.602306 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:12.602316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:12.603026 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:12.603033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:12.603257 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:12.603848 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:12.603874 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:12.605893 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:12.608527 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:12.608536 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:12.608552 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:12.608931 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:12.609252 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:12.609257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:12.609261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:12.609425 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:12.609886 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:12.609892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:12.610210 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 15:05:12.610440 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 15:05:12.610928 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":376,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
I20260925 15:05:12.613457 32077 heartbeater.cc:347] Connected to a master server at 127.22.236.62:38839
I20260925 15:05:12.613768 32077 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:12.614523 32077 heartbeater.cc:511] Master 127.22.236.62:38839 requested a full tablet report, sending...
I20260925 15:05:12.615881 31754 ts_manager.cc:194] Registered new tserver with Master: 31222a7e342841d4a00a519fb73625b5 (127.22.236.2:34215)
I20260925 15:05:12.616696 31754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:50501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:05:12.637318 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:38839
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:05:12.943537 32082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:12.943836 32082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:12.943948 32082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:12.950951 32082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:12.951046 32082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:12.951160 32082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:05:12.960345 32082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:38839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:12.961477 32082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:12.962638 32082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:12.974135 32087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:12.974319 32090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:12.974326 32088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:12.975217 32082 server_base.cc:1034] running on GCE node
I20260925 15:05:12.975739 32082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:12.976590 32082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:12.977782 32082 hybrid_clock.cc:648] HybridClock initialized: now 1790348712977716 us; error 59 us; skew 500 ppm
I20260925 15:05:12.978111 32082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:12 dist-test-slave-nsrc krb5kdc[31709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348712, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:12.980873 32082 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:05:12.982185 32082 webserver.cc:460] Webserver started at http://127.22.236.3:38123/ using document root <none> and password file <none>
I20260925 15:05:12.982795 32082 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:12.982900 32082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:12.983172 32082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:12.985976 32082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "35eb218e377f4f3c803f657abbedff95"
format_stamp: "Formatted at 2026-09-25 15:05:12 on dist-test-slave-nsrc"
server_key: "a4b1fda10cf5e76ee310be7f7a8bdf83"
server_key_iv: "3bb24e1e275211b5a61f769c1f49c4a1"
server_key_version: "encryptionkey@0"
I20260925 15:05:12.986685 32082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "35eb218e377f4f3c803f657abbedff95"
format_stamp: "Formatted at 2026-09-25 15:05:12 on dist-test-slave-nsrc"
server_key: "a4b1fda10cf5e76ee310be7f7a8bdf83"
server_key_iv: "3bb24e1e275211b5a61f769c1f49c4a1"
server_key_version: "encryptionkey@0"
I20260925 15:05:12.991349 32082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:05:12.994454 32097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:12.995602 32082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:05:12.995734 32082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "35eb218e377f4f3c803f657abbedff95"
format_stamp: "Formatted at 2026-09-25 15:05:12 on dist-test-slave-nsrc"
server_key: "a4b1fda10cf5e76ee310be7f7a8bdf83"
server_key_iv: "3bb24e1e275211b5a61f769c1f49c4a1"
server_key_version: "encryptionkey@0"
I20260925 15:05:12.995896 32082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:13.030042 32082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:13.033943 32082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:13.034191 32082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:13.035070 32082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:13.036372 32082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:13.036442 32082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:13.036535 32082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:13.036576 32082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:13.064647 32082 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:33375
I20260925 15:05:13.064690 32209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:33375 every 8 connection(s)
I20260925 15:05:13.065851 32082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:05:13.075831 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 32082
I20260925 15:05:13.076000 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:05:13.076470 23472 external_mini_cluster.cc:1428] Setting key 8e9bd78b26dfcd44c93a945550a1f5a9
Sep 25 15:05:13 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348712, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:05:13.083540 31945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:13.068078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:38589 (local address 127.22.236.62:38839)
0925 15:05:13.068256 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:13.068260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:13.069337 (+ 1077us) server_negotiation.cc:407] Connection header received
0925 15:05:13.071031 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:13.071037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:13.071118 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:13.071201 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:13.072662 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:13.073349 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:13.074798 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:13.075202 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:13.075304 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:13.078906 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:13.078912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:13.078916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:13.078953 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:13.081289 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:13.081881 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:13.081885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:13.081887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:13.081991 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:13.082317 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:13.082320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:13.082322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:13.082546 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:13.082644 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:13.083163 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 15:05:13.083303 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:05:13.084478 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:13.068569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:38589)
0925 15:05:13.069201 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:13.069236 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:13.070784 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:13.071440 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:13.071448 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:13.071914 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:13.072506 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:13.072525 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:13.073482 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:13.073486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:13.074674 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:13.074685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:13.075399 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:13.075406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:13.075649 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:13.076255 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:13.076280 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:13.078729 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:13.081387 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:13.081393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:13.081404 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:13.081794 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:13.082074 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:13.082077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:13.082079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:13.082244 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:13.082628 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:13.082633 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:13.082901 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 15:05:13.083105 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 15:05:13.083443 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":270,"threads_started":1}
I20260925 15:05:13.085805 32210 heartbeater.cc:347] Connected to a master server at 127.22.236.62:38839
I20260925 15:05:13.086125 32210 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:13.086942 32210 heartbeater.cc:511] Master 127.22.236.62:38839 requested a full tablet report, sending...
I20260925 15:05:13.088456 31754 ts_manager.cc:194] Registered new tserver with Master: 35eb218e377f4f3c803f657abbedff95 (127.22.236.3:33375)
I20260925 15:05:13.089370 31754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:38589
I20260925 15:05:13.091351 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.1.keytab.
Entry for principal oryx/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.1.keytab.
I20260925 15:05:13.141431 31943 heartbeater.cc:503] Master 127.22.236.62:38839 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.22.236.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.236.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.2.keytab.
Entry for principal oryx/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.2.keytab.
WARNING: no policy specified for oryx/127.22.236.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.236.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.3.keytab.
Entry for principal oryx/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.3.keytab.
I20260925 15:05:13.208788 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 31815
Sep 25 15:05:13 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:05:13.223891 32220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:13.209733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.22.236.1:38401)
0925 15:05:13.210482 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:13.210488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:13.210512 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:05:13.210574 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:13.210580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:13.210777 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:13.210919 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:13.212211 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:13.213232 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:13.214419 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:13.214965 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:13.215081 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:13.218811 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:13.218822 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:13.218835 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:13.218873 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:13.221430 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:13.222025 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:13.222032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:13.222034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:13.222158 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:13.222466 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:13.222470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:13.222472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:13.222775 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:13.223029 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:13.223325 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 15:05:13.223561 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":443,"threads_started":1}
I20260925 15:05:13.225119 31938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:13.305289 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 31815
I20260925 15:05:13.317835 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 31949
Sep 25 15:05:13 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:05:13.332091 32222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:13.318646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55680 (local address 127.22.236.2:34215)
0925 15:05:13.319213 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:13.319219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:13.319245 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:05:13.319302 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:13.319308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:13.319501 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:13.319631 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:13.320774 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:13.321862 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:13.323016 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:13.323550 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:13.323644 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:13.327363 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:13.327373 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:13.327387 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:13.327422 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:13.329963 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:13.330511 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:13.330517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:13.330519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:13.330626 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:13.330882 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:13.330884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:13.330886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:13.331208 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:13.331426 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:13.331607 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 15:05:13.331787 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":332,"threads_started":1}
I20260925 15:05:13.333494 32072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:13.412834 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 31949
I20260925 15:05:13.424932 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 32082
Sep 25 15:05:13 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:05:13.439368 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:13.425804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.22.236.3:33375)
0925 15:05:13.426382 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:13.426389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:13.426430 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:05:13.426488 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:13.426494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:13.426694 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:13.426821 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:13.428010 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:13.429084 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:13.430259 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:13.430791 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:13.430880 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:13.434787 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:13.434798 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:13.434811 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:13.434841 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:13.437287 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:13.437820 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:13.437826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:13.437828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:13.437934 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:13.438199 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:13.438201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:13.438203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:13.438514 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:13.438740 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:13.438914 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 15:05:13.439094 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":279,"threads_started":1}
I20260925 15:05:13.440598 32205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:13.520293 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 32082
I20260925 15:05:13.532353 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 31722
I20260925 15:05:13.533460 31784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:13.630586 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 31722
I20260925 15:05:13.647271 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:38839
--webserver_interface=127.22.236.62
--webserver_port=34743
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:38839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:05:13.952832 32227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:13.953135 32227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:13.953263 32227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:13.960381 32227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:05:13.960497 32227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:13.960572 32227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:13.960652 32227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:05:13.960708 32227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:05:13.970525 32227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:38839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:38839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=34743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:13.971778 32227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:13.973109 32227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:13.983971 32232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:13.984071 32233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:13.984040 32235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:13.984994 32227 server_base.cc:1034] running on GCE node
I20260925 15:05:13.985707 32227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:13.986989 32227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:13.988211 32227 hybrid_clock.cc:648] HybridClock initialized: now 1790348713988153 us; error 43 us; skew 500 ppm
I20260925 15:05:13.988528 32227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:13 dist-test-slave-nsrc krb5kdc[31709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348713, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:13.991293 32227 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:05:13.992638 32227 webserver.cc:460] Webserver started at http://127.22.236.62:34743/ using document root <none> and password file <none>
I20260925 15:05:13.993245 32227 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:13.993345 32227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:13.998383 32227 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 15:05:14.000928 32242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:14.002192 32227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 15:05:14.002326 32227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "ca5d787fb42d435694f0aa0e951ab2d3"
format_stamp: "Formatted at 2026-09-25 15:05:11 on dist-test-slave-nsrc"
server_key: "5c4b7a861eeacaf9c94371ab7f7d6b77"
server_key_iv: "a40e75e67a8c95e64e84b6fc27c6f15b"
server_key_version: "encryptionkey@0"
I20260925 15:05:14.003243 32227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:14.017320 32227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:14.021952 32227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:14.022164 32227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:14.037618 32227 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:38839
I20260925 15:05:14.037635 32293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:38839 every 8 connection(s)
I20260925 15:05:14.038946 32227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:05:14.043574 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 32227
I20260925 15:05:14.044339 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:38401
--local_ip_for_outbound_sockets=127.22.236.1
--tserver_master_addrs=127.22.236.62:38839
--webserver_port=35853
--webserver_interface=127.22.236.1
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.236.1
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:05:14.045598 32294 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:05:14.048904 32294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3: Bootstrap starting.
I20260925 15:05:14.067101 32294 log.cc:824] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:05:14.075172 32294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:05:14.075614 32294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3: Bootstrap complete.
I20260925 15:05:14.084192 32294 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38839 } }
I20260925 15:05:14.084997 32294 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5d787fb42d435694f0aa0e951ab2d3, State: Initialized, Role: FOLLOWER
I20260925 15:05:14.085572 32294 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38839 } }
I20260925 15:05:14.085781 32294 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:05:14.085877 32294 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:05:14.086030 32294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:05:14.089634 32294 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38839 } }
I20260925 15:05:14.090108 32294 leader_election.cc:302] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca5d787fb42d435694f0aa0e951ab2d3; no voters:
I20260925 15:05:14.090663 32294 leader_election.cc:288] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:05:14.090793 32298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:05:14.091828 32298 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [term 2 LEADER]: Becoming Leader. State: Replica: ca5d787fb42d435694f0aa0e951ab2d3, State: Running, Role: LEADER
I20260925 15:05:14.092391 32298 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38839 } }
I20260925 15:05:14.092765 32294 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:05:14.094631 32300 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca5d787fb42d435694f0aa0e951ab2d3. Latest consensus state: current_term: 2 leader_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38839 } } }
I20260925 15:05:14.095000 32300 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:05:14.095469 32299 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d787fb42d435694f0aa0e951ab2d3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38839 } } }
I20260925 15:05:14.095737 32299 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:05:14.098963 32304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:05:14.104684 32304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:05:14.106122 32304 catalog_manager.cc:1259] Loaded cluster ID: 8d5ee3a0ea764aed859865afede59f49
I20260925 15:05:14.106271 32304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:05:14.110381 32304 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:05:14.111815 32304 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ca5d787fb42d435694f0aa0e951ab2d3: Loaded TSK: 0
I20260925 15:05:14.113364 32304 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:05:14.380208 32296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:14.380482 32296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:14.380617 32296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:14.387789 32296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:14.387914 32296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:14.388062 32296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:05:14.397855 32296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.1.keytab
--principal=oryx/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:38401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=35853
--tserver_master_addrs=127.22.236.62:38839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:14.399111 32296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:14.400425 32296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:14.412995 32322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:14.413071 32321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:14.414757 32324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:14.414980 32296 server_base.cc:1034] running on GCE node
I20260925 15:05:14.415732 32296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:14.416893 32296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:14.418126 32296 hybrid_clock.cc:648] HybridClock initialized: now 1790348714418077 us; error 38 us; skew 500 ppm
I20260925 15:05:14.418491 32296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:14 dist-test-slave-nsrc krb5kdc[31709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348714, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:14.422065 32296 init.cc:377] Logged in from keytab as oryx/127.22.236.1@KRBTEST.COM (short username oryx)
I20260925 15:05:14.423422 32296 webserver.cc:460] Webserver started at http://127.22.236.1:35853/ using document root <none> and password file <none>
I20260925 15:05:14.424052 32296 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:14.424206 32296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:14.429286 32296 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:05:14.431813 32331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:14.433120 32296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:05:14.433259 32296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "f11e77284cbf47fcbae02797d55773f1"
format_stamp: "Formatted at 2026-09-25 15:05:12 on dist-test-slave-nsrc"
server_key: "23e5fa385b87ab7aa330962254933eba"
server_key_iv: "b6ba0646955defa87b54834597eddfdf"
server_key_version: "encryptionkey@0"
I20260925 15:05:14.434118 32296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:14.445977 32296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:14.449723 32296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:14.449954 32296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:14.450881 32296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:14.452134 32296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:14.452260 32296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:14.452354 32296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:14.452421 32296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:14.481451 32296 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:38401
I20260925 15:05:14.481547 32443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:38401 every 8 connection(s)
I20260925 15:05:14.482785 32296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:05:14.484100 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 32296
I20260925 15:05:14.484824 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:34215
--local_ip_for_outbound_sockets=127.22.236.2
--tserver_master_addrs=127.22.236.62:38839
--webserver_port=42627
--webserver_interface=127.22.236.2
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.236.2
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:05:14 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:14 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:05:14.799824 32448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:14.800082 32448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:14.800220 32448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:14.807231 32448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:14.807327 32448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:14.807477 32448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:05:14.817085 32448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.2.keytab
--principal=oryx/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:34215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=42627
--tserver_master_addrs=127.22.236.62:38839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:14.818300 32448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:14.819548 32448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:14.834154 32455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:14.835028 32454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:14.835418 32457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:14.836385 32448 server_base.cc:1034] running on GCE node
I20260925 15:05:14.837044 32448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:14.838070 32448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:14.839257 32448 hybrid_clock.cc:648] HybridClock initialized: now 1790348714839196 us; error 62 us; skew 500 ppm
I20260925 15:05:14.839548 32448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:14 dist-test-slave-nsrc krb5kdc[31709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348714, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:14.842710 32448 init.cc:377] Logged in from keytab as oryx/127.22.236.2@KRBTEST.COM (short username oryx)
I20260925 15:05:14.844206 32448 webserver.cc:460] Webserver started at http://127.22.236.2:42627/ using document root <none> and password file <none>
I20260925 15:05:14.844870 32448 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:14.845001 32448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:14.850051 32448 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:05:14.852533 32464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:14.853786 32448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:05:14.853917 32448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "31222a7e342841d4a00a519fb73625b5"
format_stamp: "Formatted at 2026-09-25 15:05:12 on dist-test-slave-nsrc"
server_key: "ecb6a14ec67d4fdcba8d626e1127df96"
server_key_iv: "99023a8810cba4217b5a49e6124687af"
server_key_version: "encryptionkey@0"
I20260925 15:05:14.854779 32448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:14.889941 32448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:14.894011 32448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:14.894258 32448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:14.895227 32448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:14.896595 32448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:14.896685 32448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:14.896790 32448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:14.896853 32448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:14.931736 32448 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:34215
I20260925 15:05:14.931771 32576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:34215 every 8 connection(s)
I20260925 15:05:14.933845 32448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:05:14.938479 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 32448
I20260925 15:05:14.939270 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:33375
--local_ip_for_outbound_sockets=127.22.236.3
--tserver_master_addrs=127.22.236.62:38839
--webserver_port=38123
--webserver_interface=127.22.236.3
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.236.3
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:05:14 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:14 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:05:15.457268 32581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:15.457741 32581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:15.457896 32581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:15.470824 32581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:15.470992 32581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:15.471205 32581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:05:15.488533 32581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/krb5kdc/oryx_127.22.236.3.keytab
--principal=oryx/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:33375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=38123
--tserver_master_addrs=127.22.236.62:38839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:15.490332 32581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:15.492260 32581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:15.493289 32446 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.993s user 0.000s sys 0.011s
I20260925 15:05:15.494939 32446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:14.485954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:49027)
0925 15:05:14.486702 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:14.486756 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:14.488412 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:14.493895 (+ 5483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:14.493903 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:14.494415 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:14.494954 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:14.494972 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:14.497144 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:14.497149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:14.498422 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:14.498435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:14.499722 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:14.499727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:14.499936 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:14.500669 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:14.500693 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:15.493972 (+993279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":293,"threads_started":1}
W20260925 15:05:15.495108 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:14.485978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:49027 (local address 127.22.236.62:38839)
0925 15:05:14.492639 (+ 6661us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:14.492655 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:14.492697 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:05:14.493020 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:14.493035 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:14.493401 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:14.493758 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:14.495186 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:14.496986 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:14.498567 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:14.499506 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:14.499684 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:15.494358 (+994674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:49027: BlockingRecv error: recv got EOF from 127.22.236.1:49027 (error 108)
Metrics: {"server-negotiator.queue_time_us":6422,"thread_start_us":6234,"threads_started":1}
W20260925 15:05:15.495814 32444 heartbeater.cc:644] Failed to heartbeat to 127.22.236.62:38839 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.236.62:38839: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Sep 25 15:05:15 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:15 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:15.504968 32446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:15.496481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:33637)
0925 15:05:15.496805 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:15.496823 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:15.496930 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:15.497237 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:15.497242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:15.497577 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:15.497732 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:15.497740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:15.498674 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:15.498678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:15.499617 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:15.499628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:15.500303 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:15.500309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:15.500539 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:15.501103 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:15.501127 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:15.504683 (+ 3556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205}
W20260925 15:05:15.505532 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:15.496819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:33637 (local address 127.22.236.62:38839)
0925 15:05:15.496976 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:15.496981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:15.496999 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:05:15.497054 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:15.497059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:15.497118 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:15.497197 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:15.497873 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:15.498543 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:15.499743 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:15.500156 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:15.500269 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:15.505329 (+ 5060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:33637: BlockingRecv error: recv got EOF from 127.22.236.1:33637 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 15:05:15.512272 32587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:15.513016 32588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 25 15:05:15 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:15 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:15.516211 32446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:15.505572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:37141)
0925 15:05:15.505990 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:15.506006 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:15.506094 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:15.506794 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:15.506799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:15.507169 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:15.507345 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:15.507357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:15.508741 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:15.508746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:15.510189 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:15.510202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:15.511348 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:15.511355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:15.511631 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:15.512843 (+ 1212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:15.512876 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:15.515945 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305}
W20260925 15:05:15.516372 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:15.505836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:37141 (local address 127.22.236.62:38839)
0925 15:05:15.506396 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:15.506402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:15.506416 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:05:15.506461 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:15.506466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:15.506544 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:15.506629 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:15.507509 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:15.508577 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:15.510396 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:15.510980 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:15.511136 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:15.516166 (+ 5030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:37141: BlockingRecv error: recv got EOF from 127.22.236.1:37141 (error 108)
Metrics: {"server-negotiator.queue_time_us":459}
W20260925 15:05:15.516708 32444 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:05:15.521912 32590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:15.522635 32581 server_base.cc:1034] running on GCE node
I20260925 15:05:15.523499 32581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:15.524873 32581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:15.526122 32581 hybrid_clock.cc:648] HybridClock initialized: now 1790348715526063 us; error 40 us; skew 500 ppm
I20260925 15:05:15.526687 32581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:15 dist-test-slave-nsrc krb5kdc[31709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348715, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:15.531083 32581 init.cc:377] Logged in from keytab as oryx/127.22.236.3@KRBTEST.COM (short username oryx)
I20260925 15:05:15.533164 32581 webserver.cc:460] Webserver started at http://127.22.236.3:38123/ using document root <none> and password file <none>
I20260925 15:05:15.534199 32581 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:15.534361 32581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:15.541160 32581 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:05:15.543968 32597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:15.545499 32581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:05:15.545643 32581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "35eb218e377f4f3c803f657abbedff95"
format_stamp: "Formatted at 2026-09-25 15:05:12 on dist-test-slave-nsrc"
server_key: "a4b1fda10cf5e76ee310be7f7a8bdf83"
server_key_iv: "3bb24e1e275211b5a61f769c1f49c4a1"
server_key_version: "encryptionkey@0"
I20260925 15:05:15.546550 32581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:15.568364 32581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:15.572440 32581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:15.572670 32581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:15.573627 32581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:15.574981 32581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:15.575074 32581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:15.575174 32581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:15.575239 32581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:15.607520 32581 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:33375
I20260925 15:05:15.607635 32709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:33375 every 8 connection(s)
I20260925 15:05:15.608983 32581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:05:15.613205 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 32581
Sep 25 15:05:15 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:15 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:15.645733 32713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:15.614063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.22.236.62:38839)
0925 15:05:15.629675 (+ 15612us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:15.629681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:15.630191 (+ 510us) server_negotiation.cc:407] Connection header received
0925 15:05:15.630353 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:15.630359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:15.630431 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:15.630524 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:15.631923 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:15.632699 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:15.633893 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:15.634279 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:15.634384 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:15.637053 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:15.637062 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:15.637080 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:15.637117 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:15.641421 (+ 4304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:15.642439 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:15.642449 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:15.642459 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:15.642623 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:15.642964 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:15.642969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:15.642973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:15.643616 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:15.643962 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:15.644508 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 15:05:15.645262 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15479,"thread_start_us":16266,"threads_started":1}
W20260925 15:05:16.028620 32580 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.083s user 0.002s sys 0.012s
W20260925 15:05:16.029779 32579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:14.936606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:53563 (local address 127.22.236.62:38839)
0925 15:05:14.937169 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:14.937175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:14.938143 (+ 968us) server_negotiation.cc:407] Connection header received
0925 15:05:14.940084 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:14.940089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:14.940198 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:14.940295 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:14.941744 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:14.942503 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:14.944023 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:14.944497 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:14.944616 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:16.029477 (+1084861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:53563: BlockingRecv error: recv got EOF from 127.22.236.2:53563 (error 108)
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":243,"threads_started":1}
I20260925 15:05:16.030354 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:14.937110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:53563)
0925 15:05:14.937957 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:14.938002 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:14.939825 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:14.940541 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:14.940549 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:14.941033 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:14.941578 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:14.941595 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:14.942665 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:14.942669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:14.943872 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:14.943883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:14.944725 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:14.944732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:14.944977 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:14.945600 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:14.945626 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:16.029300 (+1083674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":323,"threads_started":1}
W20260925 15:05:16.031133 32577 heartbeater.cc:644] Failed to heartbeat to 127.22.236.62:38839 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.236.62:38839: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Sep 25 15:05:16 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:16 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:16.039783 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:16.031761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:37245)
0925 15:05:16.031934 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:16.031954 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:16.032072 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:16.032383 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:16.032388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:16.032721 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:16.032869 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:16.032878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.033877 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.033881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:16.034818 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:16.034829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.035480 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.035486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:16.035623 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:16.036189 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:16.036207 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:16.039530 (+ 3323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 15:05:16.039935 32579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:16.031780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:37245 (local address 127.22.236.62:38839)
0925 15:05:16.031920 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:16.031925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:16.032019 (+ 94us) server_negotiation.cc:407] Connection header received
0925 15:05:16.032189 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:16.032195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:16.032260 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:16.032334 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:16.033000 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.033744 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.034937 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.035323 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.035425 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:16.039728 (+ 4303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:37245: BlockingRecv error: recv got EOF from 127.22.236.2:37245 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 15:05:16 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:16 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:16.048386 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:16.040596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:40067)
0925 15:05:16.040907 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:16.040924 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:16.041017 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:16.041300 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:16.041305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:16.041584 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:16.041718 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:16.041726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.042727 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.042732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:16.043891 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:16.043902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.044636 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.044642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:16.044814 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:16.045307 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:16.045322 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:16.048129 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220}
W20260925 15:05:16.048810 32577 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:05:16.048877 32579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:16.040768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:40067 (local address 127.22.236.62:38839)
0925 15:05:16.040906 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:16.040911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:16.040968 (+ 57us) server_negotiation.cc:407] Connection header received
0925 15:05:16.041111 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:16.041116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:16.041190 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:16.041260 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:16.041851 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.042599 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.044014 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.044508 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.044610 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:16.048591 (+ 3981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:40067: BlockingRecv error: recv got EOF from 127.22.236.2:40067 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 15:05:16 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:16 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:16.530848 32716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:16.517362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:60505)
0925 15:05:16.518005 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:16.518033 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:16.518191 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:16.518526 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:16.518532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:16.519106 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:16.519318 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:16.519330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.520302 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.520307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:16.521971 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:16.521987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.522676 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.522682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:16.522870 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:16.526666 (+ 3796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:16.526693 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:16.530526 (+ 3833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":357,"threads_started":1}
W20260925 15:05:16.530941 32579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:16.517457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:60505 (local address 127.22.236.62:38839)
0925 15:05:16.517664 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:16.517671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:16.518121 (+ 450us) server_negotiation.cc:407] Connection header received
0925 15:05:16.518301 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:16.518304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:16.518389 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:16.518470 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:16.519465 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.520190 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.522125 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.522509 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.522599 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:16.530744 (+ 8145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:60505: BlockingRecv error: recv got EOF from 127.22.236.1:60505 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 15:05:16.617268 32712 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.992s user 0.001s sys 0.017s
W20260925 15:05:16.618168 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:15.611915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:43591 (local address 127.22.236.62:38839)
0925 15:05:15.612114 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:15.612120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:15.613465 (+ 1345us) server_negotiation.cc:407] Connection header received
0925 15:05:15.616376 (+ 2911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:15.616384 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:15.616478 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:15.616556 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:15.619004 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:15.620333 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:15.622961 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:15.623671 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:15.623770 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:16.617984 (+994214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:43591: BlockingRecv error: recv got EOF from 127.22.236.3:43591 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:05:16.618618 32712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:15.612410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:43591)
0925 15:05:15.613273 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:15.613328 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:15.616002 (+ 2674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:15.617140 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:15.617153 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:15.617918 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:15.618773 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:15.618799 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:15.620582 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:15.620588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:15.622764 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:15.622779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:15.624177 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:15.624185 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:15.624507 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:15.625356 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:15.625393 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:16.617845 (+992452us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":315,"threads_started":1}
W20260925 15:05:16.619405 32710 heartbeater.cc:644] Failed to heartbeat to 127.22.236.62:38839 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.236.62:38839: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Sep 25 15:05:16 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:16 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:16.627761 32712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:16.619946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:51771)
0925 15:05:16.620125 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:16.620168 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:16.620258 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:16.620500 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:16.620503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:16.620800 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:16.620961 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:16.620970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.621943 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.621947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:16.622915 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:16.622926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.623591 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.623596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:16.623711 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:16.624246 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:16.624262 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:16.627484 (+ 3222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 15:05:16.627887 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:16.620017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:51771 (local address 127.22.236.62:38839)
0925 15:05:16.620245 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:16.620248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:16.620263 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:05:16.620341 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:16.620344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:16.620397 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:16.620469 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:16.621057 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.621764 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.623016 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.623426 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.623508 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:16.627679 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:51771: BlockingRecv error: recv got EOF from 127.22.236.3:51771 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 25 15:05:16 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:16 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:16.635277 32712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:16.628490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:45427)
0925 15:05:16.628630 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:16.628646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:16.628717 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:16.628938 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:16.628942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:16.629198 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:16.629334 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:16.629341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.630201 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.630204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:16.631127 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:16.631139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.631726 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.631729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:16.631843 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:16.632346 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:16.632361 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:16.635101 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:05:16.635432 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:16.628547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:45427 (local address 127.22.236.62:38839)
0925 15:05:16.628678 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:16.628681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:16.628695 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:05:16.628786 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:16.628789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:16.628836 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:16.628902 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:16.629427 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.630112 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.631237 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:16.631623 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:16.631707 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:16.635267 (+ 3560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:45427: BlockingRecv error: recv got EOF from 127.22.236.3:45427 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 15:05:16.635547 32710 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 15:05:17 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:17 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:17.057768 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:17.049357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:44895)
0925 15:05:17.049816 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:17.049831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:17.049955 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:17.050276 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:17.050280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:17.050750 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:17.050920 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:17.050930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:17.051885 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:17.051891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:17.052899 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:17.052911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:17.053571 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:17.053578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:17.053743 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:17.054289 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:17.054307 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:17.057454 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":261,"threads_started":1}
W20260925 15:05:17.057935 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:17.049416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:44895 (local address 127.22.236.62:38839)
0925 15:05:17.049613 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:17.049618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:17.049906 (+ 288us) server_negotiation.cc:407] Connection header received
0925 15:05:17.050069 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:17.050074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:17.050146 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:17.050224 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:17.051046 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:17.051761 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:17.053027 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:17.053410 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:17.053512 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:17.057703 (+ 4191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:44895: BlockingRecv error: recv got EOF from 127.22.236.2:44895 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 15:05:17 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:17 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:17.540437 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:17.531758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:47703)
0925 15:05:17.532249 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:17.532271 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:17.532408 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:17.532666 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:17.532669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:17.533043 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:17.533223 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:17.533232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:17.534121 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:17.534125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:17.535182 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:17.535198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:17.535902 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:17.535908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:17.536038 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:17.536698 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:17.536722 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:17.540091 (+ 3369us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":260,"threads_started":1}
W20260925 15:05:17.540539 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:17.531817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:47703 (local address 127.22.236.62:38839)
0925 15:05:17.531958 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:17.531962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:17.532331 (+ 369us) server_negotiation.cc:407] Connection header received
0925 15:05:17.532495 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:17.532499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:17.532558 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:17.532637 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:17.533327 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:17.534005 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:17.535311 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:17.535718 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:17.535799 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:17.540317 (+ 4518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:47703: BlockingRecv error: recv got EOF from 127.22.236.1:47703 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 15:05:17 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:17 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:17.644500 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:17.636076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:57279)
0925 15:05:17.636558 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:17.636574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:17.636682 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:17.636988 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:17.636991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:17.637374 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:17.637535 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:17.637544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:17.638587 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:17.638591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:17.639581 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:17.639596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:17.640287 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:17.640291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:17.640449 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:17.640951 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:17.640989 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:17.644190 (+ 3201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":284,"threads_started":1}
W20260925 15:05:17.644619 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:17.636192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:57279 (local address 127.22.236.62:38839)
0925 15:05:17.636383 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:17.636389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:17.636641 (+ 252us) server_negotiation.cc:407] Connection header received
0925 15:05:17.636772 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:17.636775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:17.636834 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:17.636977 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:17.637644 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:17.638466 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:17.639730 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:17.640121 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:17.640233 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:17.644382 (+ 4149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:57279: BlockingRecv error: recv got EOF from 127.22.236.3:57279 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 15:05:18 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:18 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:18.066964 32720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:18.058566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:52915)
0925 15:05:18.059010 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:18.059042 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:18.059177 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:18.059476 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:18.059481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:18.059860 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:18.060031 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:18.060041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:18.061027 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:18.061031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:18.061977 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:18.061990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:18.062662 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:18.062668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:18.062846 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:18.063378 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:18.063395 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:18.066710 (+ 3315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":243,"threads_started":1}
W20260925 15:05:18.067035 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:18.058609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:52915 (local address 127.22.236.62:38839)
0925 15:05:18.058782 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:18.058787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:18.059117 (+ 330us) server_negotiation.cc:407] Connection header received
0925 15:05:18.059282 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:18.059287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:18.059348 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:18.059423 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:18.060186 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:18.060881 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:18.062117 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:18.062522 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:18.062626 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:18.066874 (+ 4248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:52915: BlockingRecv error: recv got EOF from 127.22.236.2:52915 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 15:05:18 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:18 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:18.549371 32721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:18.541389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:36611)
0925 15:05:18.541885 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:18.541904 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:18.542044 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:18.542291 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:18.542294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:18.542635 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:18.542791 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:18.542799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:18.543719 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:18.543725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:18.544724 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:18.544737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:18.545371 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:18.545376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:18.545514 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:18.546088 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:18.546106 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:18.549151 (+ 3045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":317,"threads_started":1}
W20260925 15:05:18.549491 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:18.541539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:36611 (local address 127.22.236.62:38839)
0925 15:05:18.541709 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:18.541713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:18.541998 (+ 285us) server_negotiation.cc:407] Connection header received
0925 15:05:18.542130 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:18.542133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:18.542191 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:18.542262 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:18.542894 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:18.543563 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:18.544833 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:18.545250 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:18.545336 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:18.549313 (+ 3977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:36611: BlockingRecv error: recv got EOF from 127.22.236.1:36611 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 15:05:18 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:18 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:18.654086 32722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:18.645400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:51505)
0925 15:05:18.645794 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:18.645810 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:18.645955 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:18.646207 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:18.646210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:18.646582 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:18.646754 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:18.646762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:18.647682 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:18.647685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:18.648897 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:18.648914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:18.649919 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:18.649925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:18.650095 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:18.650677 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:18.650697 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:18.653855 (+ 3158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":215,"threads_started":1}
W20260925 15:05:18.654222 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:18.645406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:51505 (local address 127.22.236.62:38839)
0925 15:05:18.645601 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:18.645608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:18.645876 (+ 268us) server_negotiation.cc:407] Connection header received
0925 15:05:18.646047 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:18.646050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:18.646109 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:18.646181 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:18.646860 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:18.647557 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:18.649047 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:18.649611 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:18.649720 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:18.654036 (+ 4316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:51505: BlockingRecv error: recv got EOF from 127.22.236.3:51505 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 15:05:19 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:19 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:19.075835 32723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:19.067763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:57289)
0925 15:05:19.068175 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:19.068190 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:19.068293 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:19.068549 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:19.068552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:19.068878 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:19.069038 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:19.069045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:19.069949 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:19.069952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:19.070837 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:19.070847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:19.071730 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:19.071736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:19.071916 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:19.072500 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:19.072517 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:19.075598 (+ 3081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":195,"threads_started":1}
W20260925 15:05:19.075992 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:19.067836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:57289 (local address 127.22.236.62:38839)
0925 15:05:19.068037 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:19.068042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:19.068256 (+ 214us) server_negotiation.cc:407] Connection header received
0925 15:05:19.068374 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:19.068378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:19.068449 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:19.068525 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:19.069141 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:19.069855 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:19.071022 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:19.071581 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:19.071713 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:19.075782 (+ 4069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:57289: BlockingRecv error: recv got EOF from 127.22.236.2:57289 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 15:05:19 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:19 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:19.558113 32724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:19.550263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:54173)
0925 15:05:19.550746 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:19.550766 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:19.550870 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:19.551145 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:19.551148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:19.551507 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:19.551656 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:19.551665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:19.552602 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:19.552605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:19.553564 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:19.553574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:19.554146 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:19.554150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:19.554265 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:19.554820 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:19.554837 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:19.557876 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":261,"threads_started":1}
W20260925 15:05:19.558235 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:19.550320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:54173 (local address 127.22.236.62:38839)
0925 15:05:19.550492 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:19.550496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:19.550836 (+ 340us) server_negotiation.cc:407] Connection header received
0925 15:05:19.550954 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:19.550958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:19.551031 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:19.551111 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:19.551770 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:19.552512 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:19.553662 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:19.554041 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:19.554125 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:19.558051 (+ 3926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:54173: BlockingRecv error: recv got EOF from 127.22.236.1:54173 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 15:05:19 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:19 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:19.662833 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:19.654855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:38601)
0925 15:05:19.655314 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:19.655328 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:19.655445 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:19.655741 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:19.655745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:19.656100 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:19.656288 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:19.656297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:19.657239 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:19.657244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:19.658229 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:19.658240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:19.658891 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:19.658897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:19.659069 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:19.659590 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:19.659608 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:19.662514 (+ 2906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":225,"threads_started":1}
W20260925 15:05:19.662930 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:19.654934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:38601 (local address 127.22.236.62:38839)
0925 15:05:19.655122 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:19.655129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:19.655396 (+ 267us) server_negotiation.cc:407] Connection header received
0925 15:05:19.655550 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:19.655555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:19.655624 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:19.655701 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:19.656407 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:19.657118 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:19.658346 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:19.658768 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:19.658868 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:19.662708 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:38601: BlockingRecv error: recv got EOF from 127.22.236.3:38601 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 15:05:20 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:20 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:20.084694 32726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:20.076673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:40973)
0925 15:05:20.077175 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:20.077191 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:20.077300 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:20.077596 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:20.077600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:20.077965 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:20.078132 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:20.078141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:20.079063 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:20.079066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:20.080047 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:20.080058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:20.080657 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:20.080660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:20.080795 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:20.081322 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:20.081337 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:20.084451 (+ 3114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":282,"threads_started":1}
W20260925 15:05:20.084877 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:20.076736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:40973 (local address 127.22.236.62:38839)
0925 15:05:20.076886 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:20.076890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:20.077253 (+ 363us) server_negotiation.cc:407] Connection header received
0925 15:05:20.077381 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:20.077385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:20.077440 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:20.077513 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:20.078237 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:20.078956 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:20.080174 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:20.080553 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:20.080640 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:20.084640 (+ 4000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:40973: BlockingRecv error: recv got EOF from 127.22.236.2:40973 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 15:05:20 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:20 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:20.567572 32727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:20.558988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:45689)
0925 15:05:20.559486 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:20.559506 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:20.559643 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:20.559977 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:20.559982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:20.560558 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:20.560715 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:20.560727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:20.562000 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:20.562004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:20.562903 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:20.562913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:20.563542 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:20.563547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:20.563698 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:20.564245 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:20.564262 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:20.567356 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":258,"threads_started":1}
W20260925 15:05:20.567724 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:20.559116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:45689 (local address 127.22.236.62:38839)
0925 15:05:20.559319 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:20.559326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:20.559583 (+ 257us) server_negotiation.cc:407] Connection header received
0925 15:05:20.559752 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:20.559758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:20.559817 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:20.559897 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:20.560862 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:20.561873 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:20.563023 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:20.563398 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:20.563486 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:20.567529 (+ 4043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:45689: BlockingRecv error: recv got EOF from 127.22.236.1:45689 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 15:05:20 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:20 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:20.672098 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:20.663833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:58835)
0925 15:05:20.664281 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:20.664297 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:20.664406 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:20.664707 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:20.664710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:20.665080 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:20.665231 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:20.665240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:20.666507 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:20.666512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:20.667538 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:20.667550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:20.668210 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:20.668215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:20.668337 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:20.668857 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:20.668877 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:20.671865 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":220,"threads_started":1}
W20260925 15:05:20.672343 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:20.663913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:58835 (local address 127.22.236.62:38839)
0925 15:05:20.664125 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:20.664131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:20.664363 (+ 232us) server_negotiation.cc:407] Connection header received
0925 15:05:20.664537 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:20.664541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:20.664602 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:20.664674 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:20.665368 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:20.666362 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:20.667653 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:20.668032 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:20.668123 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:20.672104 (+ 3981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:58835: BlockingRecv error: recv got EOF from 127.22.236.3:58835 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 15:05:21 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:21 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:21.094864 32729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:21.085640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:41803)
0925 15:05:21.086190 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:21.086208 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:21.086342 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:21.086736 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:21.086740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:21.087308 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:21.087519 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:21.087531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:21.088798 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:21.088819 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:21.090108 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:21.090121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:21.090792 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:21.090798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:21.090980 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:21.091469 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:21.091485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:21.094571 (+ 3086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":265,"threads_started":1}
W20260925 15:05:21.095014 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:21.085800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:41803 (local address 127.22.236.62:38839)
0925 15:05:21.086001 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:21.086006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:21.086274 (+ 268us) server_negotiation.cc:407] Connection header received
0925 15:05:21.086487 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:21.086492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:21.086576 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:21.086682 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:21.087645 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:21.088660 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:21.090250 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:21.090663 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:21.090766 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:21.094779 (+ 4013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:41803: BlockingRecv error: recv got EOF from 127.22.236.2:41803 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 15:05:21 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:21 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:21.579244 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:21.568405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:49329)
0925 15:05:21.568837 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:21.568857 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:21.568963 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:21.569214 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:21.569217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:21.569549 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:21.569718 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:21.569726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:21.570618 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:21.570621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:21.571524 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:21.571534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:21.572193 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:21.572200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:21.572325 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:21.575704 (+ 3379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:21.575722 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:21.579026 (+ 3304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":234,"threads_started":1}
W20260925 15:05:21.579346 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:21.568497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:49329 (local address 127.22.236.62:38839)
0925 15:05:21.568641 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:21.568645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:21.568918 (+ 273us) server_negotiation.cc:407] Connection header received
0925 15:05:21.569044 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:21.569048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:21.569105 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:21.569186 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:21.569819 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:21.570517 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:21.571636 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:21.572046 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:21.572133 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:21.579175 (+ 7042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:49329: BlockingRecv error: recv got EOF from 127.22.236.1:49329 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 15:05:21 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:21 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:21.681427 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:21.673199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:57823)
0925 15:05:21.673670 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:21.673685 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:21.673808 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:21.674114 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:21.674119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:21.674535 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:21.674676 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:21.674685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:21.675678 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:21.675682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:21.676650 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:21.676662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:21.677336 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:21.677341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:21.677514 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:21.678097 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:21.678114 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:21.681180 (+ 3066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":258,"threads_started":1}
W20260925 15:05:21.681524 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:21.673265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:57823 (local address 127.22.236.62:38839)
0925 15:05:21.673430 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:21.673436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:21.673760 (+ 324us) server_negotiation.cc:407] Connection header received
0925 15:05:21.673916 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:21.673922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:21.673993 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:21.674069 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:21.674805 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:21.675550 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:21.676772 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:21.677206 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:21.677310 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:21.681341 (+ 4031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:57823: BlockingRecv error: recv got EOF from 127.22.236.3:57823 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 15:05:22 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:22 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:22.104424 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:22.095849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:46299)
0925 15:05:22.096451 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:22.096473 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:22.096602 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:22.097083 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:22.097088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:22.097568 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:22.097785 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:22.097799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:22.098839 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:22.098843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:22.099734 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:22.099745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:22.100379 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:22.100383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:22.100504 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:22.101039 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:22.101054 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:22.104101 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":212,"threads_started":1}
W20260925 15:05:22.104523 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:22.095905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:46299 (local address 127.22.236.62:38839)
0925 15:05:22.096729 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:22.096742 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:22.096761 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:05:22.096815 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:22.096820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:22.096894 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:22.097001 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:22.097918 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:22.098724 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:22.099857 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:22.100268 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:22.100359 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:22.104310 (+ 3951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:46299: BlockingRecv error: recv got EOF from 127.22.236.2:46299 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 15:05:22 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:22 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:22.588116 32733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:22.580104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:33939)
0925 15:05:22.580665 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:22.580685 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:22.580801 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:22.581067 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:22.581071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:22.581475 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:22.581637 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:22.581646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:22.582564 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:22.582568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:22.583534 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:22.583545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:22.584125 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:22.584129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:22.584276 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:22.584815 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:22.584838 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:22.587846 (+ 3008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":246,"threads_started":1}
W20260925 15:05:22.588219 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:22.580197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:33939 (local address 127.22.236.62:38839)
0925 15:05:22.580351 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:22.580354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:22.580751 (+ 397us) server_negotiation.cc:407] Connection header received
0925 15:05:22.580880 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:22.580883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:22.580940 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:22.581013 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:22.581749 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:22.582438 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:22.583640 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:22.584022 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:22.584100 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:22.588009 (+ 3909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:33939: BlockingRecv error: recv got EOF from 127.22.236.1:33939 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 15:05:22 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:22 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:22.690268 32734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:22.682294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:55999)
0925 15:05:22.682701 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:22.682718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:22.682826 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:22.683096 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:22.683099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:22.683466 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:22.683629 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:22.683639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:22.684618 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:22.684622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:22.685557 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:22.685568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:22.686169 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:22.686173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:22.686297 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:22.686833 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:22.686849 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:22.689967 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":208,"threads_started":1}
W20260925 15:05:22.690366 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:22.682359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:55999 (local address 127.22.236.62:38839)
0925 15:05:22.682507 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:22.682511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:22.682777 (+ 266us) server_negotiation.cc:407] Connection header received
0925 15:05:22.682926 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:22.682930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:22.682988 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:22.683057 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:22.683729 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:22.684491 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:22.685670 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:22.686045 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:22.686137 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:22.690152 (+ 4015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:55999: BlockingRecv error: recv got EOF from 127.22.236.3:55999 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 15:05:23 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:23 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:23.113585 32735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:23.105421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:49321)
0925 15:05:23.105943 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:23.105960 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:23.106078 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:23.106349 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:23.106352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:23.106803 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:23.106998 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:23.107009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:23.107934 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:23.107938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:23.108909 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:23.108921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:23.109575 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:23.109581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:23.109703 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:23.110294 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:23.110312 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:23.113342 (+ 3030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":217,"threads_started":1}
W20260925 15:05:23.113718 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:23.105521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:49321 (local address 127.22.236.62:38839)
0925 15:05:23.105785 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:23.105791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:23.106025 (+ 234us) server_negotiation.cc:407] Connection header received
0925 15:05:23.106175 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:23.106178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:23.106236 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:23.106302 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:23.107111 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:23.107791 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:23.109015 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:23.109420 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:23.109507 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:23.113527 (+ 4020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:49321: BlockingRecv error: recv got EOF from 127.22.236.2:49321 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 25 15:05:23 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:23 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:23.597467 32736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:23.588964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:40417)
0925 15:05:23.589466 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:23.589486 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:23.589622 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:23.589872 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:23.589875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:23.590261 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:23.590436 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:23.590445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:23.591320 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:23.591324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:23.592317 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:23.592330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:23.593048 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:23.593053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:23.593172 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:23.593764 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:23.593781 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:23.597207 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":243,"threads_started":1}
W20260925 15:05:23.597673 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:23.589066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:40417 (local address 127.22.236.62:38839)
0925 15:05:23.589227 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:23.589231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:23.589548 (+ 317us) server_negotiation.cc:407] Connection header received
0925 15:05:23.589707 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:23.589711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:23.589768 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:23.589840 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:23.590538 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:23.591225 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:23.592432 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:23.592862 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:23.592948 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:23.597440 (+ 4492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:40417: BlockingRecv error: recv got EOF from 127.22.236.1:40417 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 15:05:23 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:23 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:23.699258 32737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:23.691029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:32783)
0925 15:05:23.691565 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:23.691581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:23.691693 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:23.691954 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:23.691957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:23.692374 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:23.692534 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:23.692542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:23.693501 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:23.693505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:23.694492 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:23.694505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:23.695129 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:23.695133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:23.695242 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:23.695761 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:23.695778 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:23.698953 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":243,"threads_started":1}
W20260925 15:05:23.699549 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:23.691181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:32783 (local address 127.22.236.62:38839)
0925 15:05:23.691359 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:23.691365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:23.691646 (+ 281us) server_negotiation.cc:407] Connection header received
0925 15:05:23.691794 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:23.691798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:23.691855 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:23.691925 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:23.692656 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:23.693385 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:23.694591 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:23.695008 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:23.695097 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:23.699133 (+ 4036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:32783: BlockingRecv error: recv got EOF from 127.22.236.3:32783 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 15:05:24 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:24 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:24.122503 32738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:24.114490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:36925)
0925 15:05:24.114980 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:24.114996 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:24.115105 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:24.115349 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:24.115352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:24.115759 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:24.115918 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:24.115927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:24.117022 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:24.117026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:24.117932 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:24.117953 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:24.118566 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:24.118573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:24.118692 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:24.119253 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:24.119271 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:24.122267 (+ 2996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
W20260925 15:05:24.122614 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:24.114514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:36925 (local address 127.22.236.62:38839)
0925 15:05:24.114678 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:24.114683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:24.115058 (+ 375us) server_negotiation.cc:407] Connection header received
0925 15:05:24.115188 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:24.115191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:24.115248 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:24.115319 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:24.116052 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:24.116899 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:24.118044 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:24.118408 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:24.118491 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:24.122421 (+ 3930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:36925: BlockingRecv error: recv got EOF from 127.22.236.2:36925 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 15:05:24 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:24 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:24.606515 32739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:24.598534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:51235)
0925 15:05:24.598958 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:24.598979 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:24.599105 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:24.599370 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:24.599373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:24.599739 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:24.599878 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:24.599886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:24.600907 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:24.600912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:24.601840 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:24.601852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:24.602440 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:24.602444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:24.602561 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:24.603079 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:24.603095 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:24.606256 (+ 3161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":222,"threads_started":1}
W20260925 15:05:24.606654 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:24.598567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:51235 (local address 127.22.236.62:38839)
0925 15:05:24.598761 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:24.598768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:24.599051 (+ 283us) server_negotiation.cc:407] Connection header received
0925 15:05:24.599204 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:24.599208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:24.599267 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:24.599341 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:24.599984 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:24.600758 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:24.601949 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:24.602333 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:24.602415 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:24.606444 (+ 4029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:51235: BlockingRecv error: recv got EOF from 127.22.236.1:51235 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 15:05:24 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:24 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:24.708639 32740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:24.700198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:52061)
0925 15:05:24.700817 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:24.700833 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:24.700984 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:24.701249 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:24.701252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:24.701622 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:24.701798 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:24.701807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:24.702717 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:24.702721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:24.703634 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:24.703645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:24.704382 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:24.704387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:24.704507 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:24.705091 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:24.705108 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:24.708389 (+ 3281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":384,"threads_started":1}
W20260925 15:05:24.708734 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:24.700253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:52061 (local address 127.22.236.62:38839)
0925 15:05:24.700444 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:24.700451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:24.700906 (+ 455us) server_negotiation.cc:407] Connection header received
0925 15:05:24.701078 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:24.701082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:24.701141 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:24.701220 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:24.701904 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:24.702578 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:24.703735 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:24.704195 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:24.704281 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:24.708553 (+ 4272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:52061: BlockingRecv error: recv got EOF from 127.22.236.3:52061 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 15:05:25 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:25 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:25.131580 32741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:25.123301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:35169)
0925 15:05:25.123748 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:25.123763 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:25.123884 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:25.124206 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:25.124210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:25.124607 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:25.124779 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:25.124791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:25.125870 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:25.125875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:25.126941 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:25.126952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:25.127603 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:25.127608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:25.127737 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:25.128303 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:25.128325 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:25.131332 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":227,"threads_started":1}
W20260925 15:05:25.131676 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:25.123380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:35169 (local address 127.22.236.62:38839)
0925 15:05:25.123564 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:25.123570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:25.123836 (+ 266us) server_negotiation.cc:407] Connection header received
0925 15:05:25.124001 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:25.124005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:25.124062 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:25.124158 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:25.124927 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:25.125729 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:25.127055 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:25.127458 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:25.127545 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:25.131479 (+ 3934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:35169: BlockingRecv error: recv got EOF from 127.22.236.2:35169 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 15:05:25 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:25 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:25.615772 32742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:25.607388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:45359)
0925 15:05:25.607849 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:25.607869 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:25.607989 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:25.608335 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:25.608339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:25.608707 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:25.608854 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:25.608863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:25.609800 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:25.609807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:25.610745 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:25.610756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:25.611427 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:25.611433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:25.611621 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:25.612200 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:25.612217 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:25.615429 (+ 3212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":243,"threads_started":1}
W20260925 15:05:25.615845 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:25.607436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:45359 (local address 127.22.236.62:38839)
0925 15:05:25.607655 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:25.607659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:25.607945 (+ 286us) server_negotiation.cc:407] Connection header received
0925 15:05:25.608101 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:25.608106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:25.608204 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:25.608280 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:25.608985 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:25.609669 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:25.610868 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:25.611268 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:25.611368 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:25.615612 (+ 4244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:45359: BlockingRecv error: recv got EOF from 127.22.236.1:45359 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 25 15:05:25 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:25 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:25.717906 32743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:25.709583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:52437)
0925 15:05:25.710097 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:25.710112 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:25.710248 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:25.710584 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:25.710590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:25.710995 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:25.711147 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:25.711156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:25.712219 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:25.712223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:25.713226 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:25.713238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:25.713887 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:25.713893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:25.714063 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:25.714610 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:25.714627 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:25.717700 (+ 3073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":248,"threads_started":1}
W20260925 15:05:25.718120 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:25.709714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:52437 (local address 127.22.236.62:38839)
0925 15:05:25.709937 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:25.709944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:25.710190 (+ 246us) server_negotiation.cc:407] Connection header received
0925 15:05:25.710361 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:25.710366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:25.710458 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:25.710538 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:25.711278 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:25.712056 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:25.713345 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:25.713752 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:25.713866 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:25.717889 (+ 4023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:52437: BlockingRecv error: recv got EOF from 127.22.236.3:52437 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 25 15:05:26 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:26 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:26.140842 32744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:26.132399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:58329)
0925 15:05:26.132868 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:26.132883 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:26.133036 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:26.133334 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:26.133339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:26.133722 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:26.133882 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:26.133891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:26.134841 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:26.134845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:26.135803 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:26.135813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:26.136588 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:26.136599 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:26.136747 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:26.137384 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:26.137406 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:26.140595 (+ 3189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":249,"threads_started":1}
W20260925 15:05:26.140946 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:26.132489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:58329 (local address 127.22.236.62:38839)
0925 15:05:26.132708 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:26.132714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:26.132956 (+ 242us) server_negotiation.cc:407] Connection header received
0925 15:05:26.133146 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:26.133152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:26.133210 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:26.133285 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:26.134012 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:26.134715 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:26.135925 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:26.136408 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:26.136546 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:26.140772 (+ 4226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:58329: BlockingRecv error: recv got EOF from 127.22.236.2:58329 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 15:05:26 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:26 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:26.629520 32745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:26.616794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:59769)
0925 15:05:26.617267 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:26.617286 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:26.617406 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:26.617711 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:26.617716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:26.618101 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:26.618276 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:26.618285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:26.619239 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:26.619244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:26.620181 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:26.620197 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:26.620870 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:26.620876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:26.621053 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:26.626069 (+ 5016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:26.626087 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:26.629218 (+ 3131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":251,"threads_started":1}
W20260925 15:05:26.629667 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:26.616977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:59769 (local address 127.22.236.62:38839)
0925 15:05:26.617161 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:26.617166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:26.617352 (+ 186us) server_negotiation.cc:407] Connection header received
0925 15:05:26.617521 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:26.617526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:26.617583 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:26.617660 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:26.618405 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:26.619113 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:26.620316 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:26.620741 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:26.620846 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:26.629419 (+ 8573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:59769: BlockingRecv error: recv got EOF from 127.22.236.1:59769 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 15:05:26 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:26 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:26.727030 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:26.718733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:39865)
0925 15:05:26.719221 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:26.719237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:26.719346 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:26.719601 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:26.719604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:26.719959 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:26.720169 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:26.720181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:26.721061 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:26.721065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:26.722014 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:26.722025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:26.722816 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:26.722822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:26.722964 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:26.723610 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:26.723627 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:26.726805 (+ 3178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":275,"threads_started":1}
W20260925 15:05:26.727159 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:26.718787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:39865 (local address 127.22.236.62:38839)
0925 15:05:26.718939 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:26.718943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:26.719300 (+ 357us) server_negotiation.cc:407] Connection header received
0925 15:05:26.719435 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:26.719439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:26.719495 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:26.719570 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:26.720282 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:26.720948 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:26.722149 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:26.722666 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:26.722777 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:26.726961 (+ 4184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:39865: BlockingRecv error: recv got EOF from 127.22.236.3:39865 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 15:05:27 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:27 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:27.149672 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:27.141642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:59647)
0925 15:05:27.142110 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:27.142126 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:27.142248 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:27.142622 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:27.142627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:27.143088 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:27.143258 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:27.143267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:27.144233 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:27.144237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:27.145132 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:27.145144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:27.145772 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:27.145777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:27.145897 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:27.146422 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:27.146439 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:27.149445 (+ 3006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":246,"threads_started":1}
W20260925 15:05:27.149871 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:27.141730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:59647 (local address 127.22.236.62:38839)
0925 15:05:27.141899 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:27.141905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:27.142200 (+ 295us) server_negotiation.cc:407] Connection header received
0925 15:05:27.142369 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:27.142374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:27.142468 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:27.142557 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:27.143368 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:27.144110 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:27.145238 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:27.145654 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:27.145736 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:27.149643 (+ 3907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:59647: BlockingRecv error: recv got EOF from 127.22.236.2:59647 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 15:05:27 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:27 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:27.638793 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:27.630439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:34833)
0925 15:05:27.630923 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:27.630943 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:27.631050 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:27.631299 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:27.631302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:27.631691 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:27.631851 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:27.631859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:27.632931 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:27.632935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:27.633857 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:27.633869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:27.634473 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:27.634477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:27.634614 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:27.635151 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:27.635168 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:27.638550 (+ 3382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":282,"threads_started":1}
W20260925 15:05:27.638913 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:27.630514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:34833 (local address 127.22.236.62:38839)
0925 15:05:27.630672 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:27.630676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:27.631011 (+ 335us) server_negotiation.cc:407] Connection header received
0925 15:05:27.631135 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:27.631138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:27.631196 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:27.631272 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:27.631956 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:27.632800 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:27.633977 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:27.634369 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:27.634466 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:27.638714 (+ 4248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:34833: BlockingRecv error: recv got EOF from 127.22.236.1:34833 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 15:05:27 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:27 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:27.735899 32749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:27.727854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:49985)
0925 15:05:27.728346 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:27.728365 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:27.728503 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:27.728828 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:27.728832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:27.729313 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:27.729503 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:27.729513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:27.730527 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:27.730532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:27.731413 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:27.731424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:27.732079 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:27.732085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:27.732277 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:27.732776 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:27.732793 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:27.735687 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":291,"threads_started":1}
W20260925 15:05:27.736037 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:27.727940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:49985 (local address 127.22.236.62:38839)
0925 15:05:27.728215 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:27.728221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:27.728449 (+ 228us) server_negotiation.cc:407] Connection header received
0925 15:05:27.728619 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:27.728624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:27.728694 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:27.728779 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:27.729654 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:27.730400 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:27.731551 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:27.731942 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:27.732051 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:27.735856 (+ 3805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:49985: BlockingRecv error: recv got EOF from 127.22.236.3:49985 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 15:05:28 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:28 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:28.158622 32750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:28.150422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:42909)
0925 15:05:28.150993 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:28.151012 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:28.151135 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:28.151399 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:28.151402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:28.151768 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:28.151930 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:28.151938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:28.152850 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:28.152854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:28.153783 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:28.153793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:28.154452 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:28.154456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:28.154579 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:28.155088 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:28.155104 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:28.158333 (+ 3229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":299,"threads_started":1}
W20260925 15:05:28.158768 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:28.150459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:42909 (local address 127.22.236.62:38839)
0925 15:05:28.150622 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:28.150626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:28.151091 (+ 465us) server_negotiation.cc:407] Connection header received
0925 15:05:28.151223 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:28.151226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:28.151296 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:28.151371 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:28.152057 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:28.152746 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:28.153896 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:28.154320 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:28.154405 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:28.158573 (+ 4168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:42909: BlockingRecv error: recv got EOF from 127.22.236.2:42909 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 15:05:28 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:28 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:28.647738 32751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:28.639488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:32875)
0925 15:05:28.639921 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:28.639938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:28.640080 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:28.640397 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:28.640402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:28.640770 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:28.640919 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:28.640927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:28.641914 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:28.641918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:28.642800 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:28.642811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:28.643481 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:28.643486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:28.643643 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:28.644154 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:28.644171 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:28.647512 (+ 3341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":233,"threads_started":1}
W20260925 15:05:28.647917 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:28.639565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:32875 (local address 127.22.236.62:38839)
0925 15:05:28.639751 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:28.639757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:28.640012 (+ 255us) server_negotiation.cc:407] Connection header received
0925 15:05:28.640203 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:28.640208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:28.640278 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:28.640355 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:28.641052 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:28.641785 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:28.642927 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:28.643353 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:28.643453 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:28.647694 (+ 4241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:32875: BlockingRecv error: recv got EOF from 127.22.236.1:32875 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 15:05:28 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:28 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:28.745366 32752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:28.736801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:55675)
0925 15:05:28.737287 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:28.737303 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:28.737411 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:28.737758 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:28.737762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:28.738208 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:28.738380 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:28.738389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:28.739422 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:28.739427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:28.740708 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:28.740720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:28.741352 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:28.741356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:28.741504 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:28.742053 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:28.742073 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:28.745129 (+ 3056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":282,"threads_started":1}
W20260925 15:05:28.745468 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:28.736891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:55675 (local address 127.22.236.62:38839)
0925 15:05:28.737096 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:28.737101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:28.737371 (+ 270us) server_negotiation.cc:407] Connection header received
0925 15:05:28.737543 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:28.737549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:28.737625 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:28.737699 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:28.738487 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:28.739286 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:28.740813 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:28.741205 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:28.741293 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:28.745296 (+ 4003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:55675: BlockingRecv error: recv got EOF from 127.22.236.3:55675 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 15:05:29 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:29 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:29.167438 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:29.159477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:47359)
0925 15:05:29.160062 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:29.160081 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:29.160228 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:29.160510 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:29.160513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:29.160898 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:29.161034 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:29.161042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:29.161954 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:29.161957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:29.162830 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:29.162842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:29.163504 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:29.163508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:29.163627 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:29.164189 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:29.164206 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:29.167231 (+ 3025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":396,"threads_started":1}
W20260925 15:05:29.167594 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:29.159643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:47359 (local address 127.22.236.62:38839)
0925 15:05:29.159817 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:29.159823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:29.160170 (+ 347us) server_negotiation.cc:407] Connection header received
0925 15:05:29.160345 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:29.160349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:29.160408 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:29.160487 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:29.161140 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:29.161845 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:29.162938 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:29.163392 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:29.163484 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:29.167389 (+ 3905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:47359: BlockingRecv error: recv got EOF from 127.22.236.2:47359 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 15:05:29 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:29 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.1@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:29.656916 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:29.648807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.1:33851)
0925 15:05:29.649311 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:29.649335 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:29.649465 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:29.649785 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:29.649791 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:29.650235 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:29.650425 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:29.650435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:29.651337 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:29.651341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:29.652291 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:29.652302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:29.652979 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:29.652982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:29.653105 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:29.653660 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:29.653677 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:29.656688 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":255,"threads_started":1}
W20260925 15:05:29.657073 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:29.648895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:33851 (local address 127.22.236.62:38839)
0925 15:05:29.649080 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:29.649086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:29.649411 (+ 325us) server_negotiation.cc:407] Connection header received
0925 15:05:29.649573 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:29.649578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:29.649650 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:29.649745 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:29.650550 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:29.651212 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:29.652425 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:29.652861 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:29.652962 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:29.656870 (+ 3908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.1:33851: BlockingRecv error: recv got EOF from 127.22.236.1:33851 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 15:05:29 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:29 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.3@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:29.754635 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:29.746198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.3:42917)
0925 15:05:29.746653 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:29.746668 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:29.746781 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:29.747121 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:29.747124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:29.747502 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:29.747642 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:29.747651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:29.748735 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:29.748739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:29.749702 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:29.749718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:29.750435 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:29.750439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:29.750562 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:29.751108 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:29.751126 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:29.754352 (+ 3226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":250,"threads_started":1}
W20260925 15:05:29.754812 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:29.746313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:42917 (local address 127.22.236.62:38839)
0925 15:05:29.746577 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:29.746581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:29.746744 (+ 163us) server_negotiation.cc:407] Connection header received
0925 15:05:29.746873 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:29.746876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:29.746975 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:29.747065 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:29.747784 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:29.748582 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:29.749862 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:29.750291 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:29.750411 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:29.754551 (+ 4140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.3:42917: BlockingRecv error: recv got EOF from 127.22.236.3:42917 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
Sep 25 15:05:30 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:05:30 dist-test-slave-nsrc krb5kdc[31709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.236.2@KRBTEST.COM for oryx/127.22.236.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:05:30.178814 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:30.168483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:38839 (local address 127.22.236.2:49293)
0925 15:05:30.168943 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:30.168958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:30.169074 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:30.169368 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:30.169372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:30.169738 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:30.169907 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:30.169917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:30.171160 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:30.171165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:30.172556 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:30.172569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:30.173790 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:30.173795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:30.173961 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:30.174604 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:30.174626 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:30.178493 (+ 3867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.236.62:38839: Server oryx/127.22.236.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":224,"threads_started":1}
W20260925 15:05:30.179106 32447 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:30.168553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:49293 (local address 127.22.236.62:38839)
0925 15:05:30.168720 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:30.168726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:30.169037 (+ 311us) server_negotiation.cc:407] Connection header received
0925 15:05:30.169180 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:30.169184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:30.169252 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:30.169327 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:30.170045 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:30.170988 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:30.172842 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:30.173488 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:30.173576 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:30.178866 (+ 5290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.236.2:49293: BlockingRecv error: recv got EOF from 127.22.236.2:49293 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:05:30.623044 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 32296
I20260925 15:05:30.634378 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:30.623997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44640 (local address 127.22.236.1:38401)
0925 15:05:30.624610 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:30.624618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:30.624641 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:05:30.624701 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:30.624707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:30.624961 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:30.625121 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:30.626399 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:30.627461 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:30.628624 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:30.629149 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:30.629246 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:30.631594 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:30.631603 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:30.631617 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:30.631658 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:30.633917 (+ 2259us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.236.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 15:05:30.634179 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44640: Request ticket server kudu/127.22.236.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":307,"threads_started":1}
W20260925 15:05:30.634482 32758 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44640: Request ticket server kudu/127.22.236.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 15:05:30.634516 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 32296: Not authorized: Client connection negotiation failed: client connection to 127.22.236.1:38401: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.236.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 15:05:30.634721 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 32296
I20260925 15:05:30.647727 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 32448
I20260925 15:05:30.659981 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:30.648744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.22.236.2:34215)
0925 15:05:30.649370 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:30.649379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:30.649414 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:05:30.649537 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:30.649547 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:30.649905 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:30.650138 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:30.651372 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:30.652452 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:30.653805 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:30.654373 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:30.654468 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:30.656935 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:30.656944 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:30.656959 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:30.656998 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:30.659479 (+ 2481us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.236.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 15:05:30.659753 (+ 274us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38416: Request ticket server kudu/127.22.236.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":299,"threads_started":1}
W20260925 15:05:30.660058 32759 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38416: Request ticket server kudu/127.22.236.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 15:05:30.660125 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 32448: Not authorized: Client connection negotiation failed: client connection to 127.22.236.2:34215: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.236.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 15:05:30.660284 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 32448
I20260925 15:05:30.672955 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 32581
I20260925 15:05:30.684628 32760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:30.674046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50202 (local address 127.22.236.3:33375)
0925 15:05:30.674644 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:30.674650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:30.674674 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:05:30.674733 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:30.674739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:30.674966 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:30.675144 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:30.676459 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:30.677456 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:30.678631 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:30.679165 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:30.679279 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:30.681686 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:30.681695 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:30.681709 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:30.681742 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:30.683967 (+ 2225us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.236.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 15:05:30.684306 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50202: Request ticket server kudu/127.22.236.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":282,"threads_started":1}
W20260925 15:05:30.684733 32760 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50202: Request ticket server kudu/127.22.236.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 15:05:30.684725 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 32581: Not authorized: Client connection negotiation failed: client connection to 127.22.236.3:33375: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.236.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 15:05:30.684873 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 32581
I20260925 15:05:30.698287 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 32227
I20260925 15:05:30.699465 32289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:30.810220 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 32227
2026-09-25T15:05:30Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20869 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T15:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:05:30Z Disabled control of system clock
I20260925 15:05:30.874553 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:42037
--webserver_interface=127.22.236.62
--webserver_port=0
--builtin_ntp_servers=127.22.236.20:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:42037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 15:05:31.180850 32765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:31.181129 32765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:31.181218 32765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:31.188133 32765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:05:31.188244 32765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:31.188285 32765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:31.188338 32765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:05:31.188380 32765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:05:31.197081 32765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:42037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:42037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:31.198223 32765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:31.199818 32765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:31.210532 303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:31.210649 302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:31.210556 305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:31.211292 32765 server_base.cc:1034] running on GCE node
I20260925 15:05:31.211970 32765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:31.213253 32765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:31.214445 32765 hybrid_clock.cc:648] HybridClock initialized: now 1790348731214392 us; error 44 us; skew 500 ppm
I20260925 15:05:31.214761 32765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:05:31.216969 32765 webserver.cc:460] Webserver started at http://127.22.236.62:37131/ using document root <none> and password file <none>
I20260925 15:05:31.217588 32765 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:31.217684 32765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:31.217980 32765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:31.220678 32765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "c03440e26af84463b051aae7812cfeea"
format_stamp: "Formatted at 2026-09-25 15:05:31 on dist-test-slave-nsrc"
server_key: "4e733ee4b5511a5c837166189a09eed4"
server_key_iv: "cf270a28a0c9a66decc0467c63224a3e"
server_key_version: "encryptionkey@0"
I20260925 15:05:31.221380 32765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "c03440e26af84463b051aae7812cfeea"
format_stamp: "Formatted at 2026-09-25 15:05:31 on dist-test-slave-nsrc"
server_key: "4e733ee4b5511a5c837166189a09eed4"
server_key_iv: "cf270a28a0c9a66decc0467c63224a3e"
server_key_version: "encryptionkey@0"
I20260925 15:05:31.225939 32765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:05:31.229046 311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:31.230226 32765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:05:31.230371 32765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "c03440e26af84463b051aae7812cfeea"
format_stamp: "Formatted at 2026-09-25 15:05:31 on dist-test-slave-nsrc"
server_key: "4e733ee4b5511a5c837166189a09eed4"
server_key_iv: "cf270a28a0c9a66decc0467c63224a3e"
server_key_version: "encryptionkey@0"
I20260925 15:05:31.230554 32765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:31.246990 32765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:31.247728 32765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:31.247952 32765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:31.264647 32765 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:42037
I20260925 15:05:31.264665 362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:42037 every 8 connection(s)
I20260925 15:05:31.265934 32765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:05:31.269516 363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:31.271759 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 32765
I20260925 15:05:31.271937 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:05:31.272706 23472 external_mini_cluster.cc:1428] Setting key 645914ce9f7b3076a95b4c32b023c4fe
I20260925 15:05:31.280485 363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea: Bootstrap starting.
I20260925 15:05:31.283376 363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:31.284412 363 log.cc:824] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea: Log is configured to *not* fsync() on all Append() calls
I20260925 15:05:31.285480 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:31.274833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39372 (local address 127.22.236.62:42037)
0925 15:05:31.275722 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:31.275732 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:31.275772 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:05:31.277356 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:31.277379 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:31.277833 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:31.278184 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:31.278732 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:31.279903 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:31.281214 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:31.281876 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:31.282057 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:31.283116 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:31.283123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:31.283136 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:05:31.283674 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:31.283767 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:05:31.283790 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:31.283881 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:31.284087 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 15:05:31.284384 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":309,"threads_started":1}
I20260925 15:05:31.287103 363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea: No bootstrap required, opened a new log
I20260925 15:05:31.294287 363 raft_consensus.cc:348] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03440e26af84463b051aae7812cfeea" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42037 } }
I20260925 15:05:31.294682 363 raft_consensus.cc:374] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:31.294775 363 raft_consensus.cc:729] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03440e26af84463b051aae7812cfeea, State: Initialized, Role: FOLLOWER
I20260925 15:05:31.295300 363 consensus_queue.cc:260] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03440e26af84463b051aae7812cfeea" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42037 } }
I20260925 15:05:31.295528 363 raft_consensus.cc:388] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:05:31.295658 363 raft_consensus.cc:482] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:05:31.295830 363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:31.297981 363 raft_consensus.cc:504] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03440e26af84463b051aae7812cfeea" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42037 } }
I20260925 15:05:31.298445 363 leader_election.cc:302] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c03440e26af84463b051aae7812cfeea; no voters:
I20260925 15:05:31.298988 363 leader_election.cc:288] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:05:31.299101 368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:05:31.299734 368 raft_consensus.cc:686] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [term 1 LEADER]: Becoming Leader. State: Replica: c03440e26af84463b051aae7812cfeea, State: Running, Role: LEADER
I20260925 15:05:31.300261 368 consensus_queue.cc:237] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03440e26af84463b051aae7812cfeea" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42037 } }
I20260925 15:05:31.300868 363 sys_catalog.cc:564] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:05:31.302904 370 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [sys.catalog]: SysCatalogTable state changed. Reason: New leader c03440e26af84463b051aae7812cfeea. Latest consensus state: current_term: 1 leader_uuid: "c03440e26af84463b051aae7812cfeea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03440e26af84463b051aae7812cfeea" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42037 } } }
I20260925 15:05:31.303254 370 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [sys.catalog]: This master's current role is: LEADER
I20260925 15:05:31.302904 369 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c03440e26af84463b051aae7812cfeea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03440e26af84463b051aae7812cfeea" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42037 } } }
I20260925 15:05:31.303759 369 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea [sys.catalog]: This master's current role is: LEADER
I20260925 15:05:31.308909 375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:05:31.313820 375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:05:31.321801 375 catalog_manager.cc:1347] Generated new cluster ID: 62ceccd1021c4f3dbf7619c4f84664c6
I20260925 15:05:31.321959 375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:05:31.343537 375 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:05:31.344815 375 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:05:31.350767 375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c03440e26af84463b051aae7812cfeea: Generated new TSK 0
I20260925 15:05:31.351553 375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:05:31.366621 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:42037
--builtin_ntp_servers=127.22.236.20:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 15:05:31.673750 387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:31.674037 387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:31.674139 387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:31.681010 387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:31.681093 387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:31.681195 387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:05:31.690127 387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:42037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:31.691785 387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:31.693671 387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:31.705678 392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:31.705714 393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:31.705621 395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:31.706451 387 server_base.cc:1034] running on GCE node
I20260925 15:05:31.706949 387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:31.707743 387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:31.708914 387 hybrid_clock.cc:648] HybridClock initialized: now 1790348731708857 us; error 49 us; skew 500 ppm
I20260925 15:05:31.709164 387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:05:31.717198 387 webserver.cc:460] Webserver started at http://127.22.236.1:35007/ using document root <none> and password file <none>
I20260925 15:05:31.717811 387 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:31.717890 387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:31.718150 387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:31.720837 387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "8d8653dd332741a19316a52e289ead27"
format_stamp: "Formatted at 2026-09-25 15:05:31 on dist-test-slave-nsrc"
server_key: "ff1cb2285f2de79ca2f79db1e5a5bc01"
server_key_iv: "64f64c440dcf503b28f4434ee0bf128f"
server_key_version: "encryptionkey@0"
I20260925 15:05:31.721557 387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "8d8653dd332741a19316a52e289ead27"
format_stamp: "Formatted at 2026-09-25 15:05:31 on dist-test-slave-nsrc"
server_key: "ff1cb2285f2de79ca2f79db1e5a5bc01"
server_key_iv: "64f64c440dcf503b28f4434ee0bf128f"
server_key_version: "encryptionkey@0"
I20260925 15:05:31.726202 387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 15:05:31.729409 401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:31.730582 387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:05:31.730706 387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "8d8653dd332741a19316a52e289ead27"
format_stamp: "Formatted at 2026-09-25 15:05:31 on dist-test-slave-nsrc"
server_key: "ff1cb2285f2de79ca2f79db1e5a5bc01"
server_key_iv: "64f64c440dcf503b28f4434ee0bf128f"
server_key_version: "encryptionkey@0"
I20260925 15:05:31.730860 387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:31.753400 387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:31.754141 387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:31.754383 387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:31.755189 387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:31.756568 387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:31.756649 387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:31.756724 387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:31.756779 387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:31.785190 387 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:33169
I20260925 15:05:31.785255 513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:33169 every 8 connection(s)
I20260925 15:05:31.786402 387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:05:31.796373 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 387
I20260925 15:05:31.796535 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:05:31.797272 23472 external_mini_cluster.cc:1428] Setting key d53698027507cdb688ddb79bcf8f962b
I20260925 15:05:31.797688 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:31.788784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:35647 (local address 127.22.236.62:42037)
0925 15:05:31.789222 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:31.789226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:31.790338 (+ 1112us) server_negotiation.cc:407] Connection header received
0925 15:05:31.791914 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:31.791918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:31.791995 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:31.792084 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:31.793068 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:31.793772 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:31.795177 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:31.795550 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:31.795638 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:31.796977 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:31.796981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:31.796984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:05:31.797002 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:31.797041 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:05:31.797053 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:31.797119 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:31.797371 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 15:05:31.797449 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":223,"threads_started":1}
I20260925 15:05:31.798847 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:31.789365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:42037 (local address 127.22.236.1:35647)
0925 15:05:31.790182 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:31.790226 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:31.791692 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:31.792308 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:31.792316 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:31.792405 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:31.792920 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:31.792936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:31.793889 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:31.793892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:31.795065 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:31.795076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:31.795712 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:31.795716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:31.795917 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:31.796658 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:05:31.796661 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:31.796711 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:05:31.796808 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:31.797151 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:31.797161 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:31.797169 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:05:31.797344 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 15:05:31.797702 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":239,"threads_started":1}
I20260925 15:05:31.799973 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:42037
--builtin_ntp_servers=127.22.236.20:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 15:05:31.800282 514 heartbeater.cc:347] Connected to a master server at 127.22.236.62:42037
I20260925 15:05:31.800532 514 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:31.801213 514 heartbeater.cc:511] Master 127.22.236.62:42037 requested a full tablet report, sending...
I20260925 15:05:31.803138 328 ts_manager.cc:194] Registered new tserver with Master: 8d8653dd332741a19316a52e289ead27 (127.22.236.1:33169)
I20260925 15:05:31.805011 328 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.236.1:35647
W20260925 15:05:32.102581 518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:32.102834 518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:32.102948 518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:32.110446 518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:32.110584 518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:32.110749 518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:05:32.119508 518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:42037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:32.120694 518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:32.122320 518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:32.133747 524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:32.133800 523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:32.133879 526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:32.134852 518 server_base.cc:1034] running on GCE node
I20260925 15:05:32.135423 518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:32.136310 518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:32.137493 518 hybrid_clock.cc:648] HybridClock initialized: now 1790348732137451 us; error 34 us; skew 500 ppm
I20260925 15:05:32.137763 518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:05:32.139699 518 webserver.cc:460] Webserver started at http://127.22.236.2:43205/ using document root <none> and password file <none>
I20260925 15:05:32.140357 518 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:32.140444 518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:32.140728 518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:32.143307 518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "87652eddd815479dbf702e380449808c"
format_stamp: "Formatted at 2026-09-25 15:05:32 on dist-test-slave-nsrc"
server_key: "2dbd18e130896a15775f0b6d4345a88b"
server_key_iv: "5560bf77bbbd3dac7a9d88e015c4c0e3"
server_key_version: "encryptionkey@0"
I20260925 15:05:32.144241 518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "87652eddd815479dbf702e380449808c"
format_stamp: "Formatted at 2026-09-25 15:05:32 on dist-test-slave-nsrc"
server_key: "2dbd18e130896a15775f0b6d4345a88b"
server_key_iv: "5560bf77bbbd3dac7a9d88e015c4c0e3"
server_key_version: "encryptionkey@0"
I20260925 15:05:32.148809 518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 15:05:32.151839 532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:32.153012 518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:05:32.153154 518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "87652eddd815479dbf702e380449808c"
format_stamp: "Formatted at 2026-09-25 15:05:32 on dist-test-slave-nsrc"
server_key: "2dbd18e130896a15775f0b6d4345a88b"
server_key_iv: "5560bf77bbbd3dac7a9d88e015c4c0e3"
server_key_version: "encryptionkey@0"
I20260925 15:05:32.153329 518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:32.163389 518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:32.164108 518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:32.164393 518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:32.165289 518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:32.166540 518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:32.166630 518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:32.166728 518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:32.166800 518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:32.195029 518 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:34665
I20260925 15:05:32.195065 644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:34665 every 8 connection(s)
I20260925 15:05:32.196286 518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:05:32.198609 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 518
I20260925 15:05:32.198746 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:05:32.199143 23472 external_mini_cluster.cc:1428] Setting key 079732cb1aa3403f5d752147696f82a1
I20260925 15:05:32.201627 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:42037
--builtin_ntp_servers=127.22.236.20:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 15:05:32.207792 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:32.198595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:51365 (local address 127.22.236.62:42037)
0925 15:05:32.198799 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:32.198806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:32.200277 (+ 1471us) server_negotiation.cc:407] Connection header received
0925 15:05:32.201990 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:32.201996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:32.202067 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:32.202146 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:32.203162 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:32.203847 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:32.205381 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:32.205767 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:32.205873 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:32.207073 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:32.207078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:32.207081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:05:32.207097 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:32.207126 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:05:32.207137 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:32.207207 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:32.207402 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 15:05:32.207492 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:05:32.208830 647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:32.199112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:42037 (local address 127.22.236.2:51365)
0925 15:05:32.200098 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:32.200177 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:32.201705 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:32.202365 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:32.202373 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:32.202471 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:32.203008 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:32.203025 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:32.203995 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:32.203999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:32.205253 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:32.205264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:32.205966 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:32.205974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:32.206179 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:32.206795 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:05:32.206798 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:32.206833 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:05:32.206906 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:32.207219 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:32.207230 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:32.207237 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:05:32.207380 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 15:05:32.207751 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":299,"threads_started":1}
I20260925 15:05:32.210035 645 heartbeater.cc:347] Connected to a master server at 127.22.236.62:42037
I20260925 15:05:32.210342 645 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:32.211122 645 heartbeater.cc:511] Master 127.22.236.62:42037 requested a full tablet report, sending...
I20260925 15:05:32.212448 328 ts_manager.cc:194] Registered new tserver with Master: 87652eddd815479dbf702e380449808c (127.22.236.2:34665)
I20260925 15:05:32.213204 328 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.236.2:51365
W20260925 15:05:32.516316 648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:32.516595 648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:32.516728 648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:32.523777 648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:32.523880 648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:32.524003 648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:05:32.532787 648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:42037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:32.534044 648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:32.535830 648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:32.547879 653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:32.547914 654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:32.547838 656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:32.548393 648 server_base.cc:1034] running on GCE node
I20260925 15:05:32.548980 648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:32.549857 648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:32.551043 648 hybrid_clock.cc:648] HybridClock initialized: now 1790348732550995 us; error 38 us; skew 500 ppm
I20260925 15:05:32.551342 648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:05:32.553421 648 webserver.cc:460] Webserver started at http://127.22.236.3:46379/ using document root <none> and password file <none>
I20260925 15:05:32.554054 648 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:32.554159 648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:32.554452 648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:32.557255 648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "975745182fac4c5ba7e874fe0ba2d276"
format_stamp: "Formatted at 2026-09-25 15:05:32 on dist-test-slave-nsrc"
server_key: "d25f00fb00eb6b909efe027b2376ee57"
server_key_iv: "52378794992c7b006338853ec8a1c3bf"
server_key_version: "encryptionkey@0"
I20260925 15:05:32.557992 648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "975745182fac4c5ba7e874fe0ba2d276"
format_stamp: "Formatted at 2026-09-25 15:05:32 on dist-test-slave-nsrc"
server_key: "d25f00fb00eb6b909efe027b2376ee57"
server_key_iv: "52378794992c7b006338853ec8a1c3bf"
server_key_version: "encryptionkey@0"
I20260925 15:05:32.562739 648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:05:32.566049 662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:32.567227 648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:05:32.567368 648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "975745182fac4c5ba7e874fe0ba2d276"
format_stamp: "Formatted at 2026-09-25 15:05:32 on dist-test-slave-nsrc"
server_key: "d25f00fb00eb6b909efe027b2376ee57"
server_key_iv: "52378794992c7b006338853ec8a1c3bf"
server_key_version: "encryptionkey@0"
I20260925 15:05:32.567553 648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:32.590677 648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:32.591457 648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:32.591723 648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:32.592620 648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:32.593931 648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:32.594020 648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:32.594127 648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:32.594195 648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:32.621786 648 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:34651
I20260925 15:05:32.621848 774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:34651 every 8 connection(s)
I20260925 15:05:32.622960 648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:05:32.630671 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 648
I20260925 15:05:32.630806 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:05:32.631217 23472 external_mini_cluster.cc:1428] Setting key f8752ad12ac141bab4d42851095cc47d
I20260925 15:05:32.634697 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:32.625267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:52189 (local address 127.22.236.62:42037)
0925 15:05:32.625434 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:32.625438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:32.626737 (+ 1299us) server_negotiation.cc:407] Connection header received
0925 15:05:32.628416 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:32.628422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:32.628498 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:32.628583 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:32.629587 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:32.630284 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:32.631908 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:32.632480 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:32.632607 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:32.633886 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:32.633893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:32.633897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:05:32.633916 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:32.633951 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:05:32.633967 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:32.634053 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:32.634249 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 15:05:32.634336 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:05:32.635572 777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:32.625761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:42037 (local address 127.22.236.3:52189)
0925 15:05:32.626589 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:32.626625 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:32.628163 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:32.628813 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:32.628821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:32.628912 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:32.629428 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:32.629444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:32.630438 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:32.630442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:32.631772 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:32.631788 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:32.632670 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:32.632676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:32.632897 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:32.633593 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:05:32.633595 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:32.633632 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:05:32.633719 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:32.634067 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:32.634078 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:32.634086 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:05:32.634229 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 15:05:32.634589 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":257,"threads_started":1}
I20260925 15:05:32.636703 775 heartbeater.cc:347] Connected to a master server at 127.22.236.62:42037
I20260925 15:05:32.636931 775 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:32.637549 775 heartbeater.cc:511] Master 127.22.236.62:42037 requested a full tablet report, sending...
I20260925 15:05:32.638693 328 ts_manager.cc:194] Registered new tserver with Master: 975745182fac4c5ba7e874fe0ba2d276 (127.22.236.3:34651)
I20260925 15:05:32.639398 328 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.236.3:52189
I20260925 15:05:32.645850 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:05:32.651260 516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:32.649892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.22.236.62:42037)
0925 15:05:32.650083 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:32.650089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:32.650440 (+ 351us) server_negotiation.cc:407] Connection header received
0925 15:05:32.650559 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:32.650562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:32.650631 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:32.650704 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:32.651004 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39382: BlockingRecv error: recv got EOF from 127.0.0.1:39382 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:05:32.651664 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 387
I20260925 15:05:32.658790 784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:32.652651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.22.236.1:33169)
0925 15:05:32.653098 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:32.653104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:32.653131 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:05:32.653187 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:32.653193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:32.653384 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:32.653496 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:32.653823 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:32.654773 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:32.655962 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:32.656552 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:32.656647 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:32.657566 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:32.657573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:32.657585 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:05:32.658171 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:32.658214 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:05:32.658238 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:32.658316 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:32.658422 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 15:05:32.658520 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":213,"threads_started":1}
I20260925 15:05:32.659957 509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:32.739848 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 387
I20260925 15:05:32.752924 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 518
I20260925 15:05:32.760406 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:32.753731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.22.236.2:34665)
0925 15:05:32.754189 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:32.754196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:32.754220 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:05:32.754276 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:32.754282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:32.754487 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:32.754638 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:32.754996 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:32.756179 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:32.757436 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:32.757975 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:32.758084 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:32.759019 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:32.759027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:32.759040 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:05:32.759589 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:32.759633 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:05:32.759651 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:32.759759 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:32.759887 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 15:05:32.760016 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":215,"threads_started":1}
I20260925 15:05:32.761528 640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:32.839929 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 518
I20260925 15:05:32.853675 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 648
I20260925 15:05:32.861186 788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:32.854563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.22.236.3:34651)
0925 15:05:32.855177 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:32.855184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:32.855214 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:05:32.855269 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:32.855275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:32.855499 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:32.855618 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:32.856030 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:32.857085 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:32.858225 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:32.858761 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:32.858863 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:32.859732 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:32.859740 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:32.859753 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:05:32.860478 (+ 725us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:32.860523 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:05:32.860542 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:32.860645 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:32.860733 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 15:05:32.860835 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":293,"threads_started":1}
I20260925 15:05:32.862301 770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:32.941128 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 648
I20260925 15:05:32.953487 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 32765
I20260925 15:05:32.954564 358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:33.050589 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 32765
2026-09-25T15:05:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2236 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T15:05:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:05:33Z Disabled control of system clock
I20260925 15:05:33.112979 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:43407
--webserver_interface=127.22.236.62
--webserver_port=0
--builtin_ntp_servers=127.22.236.20:36975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:43407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:05:33.416921 794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:33.417172 794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:33.417259 794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:33.424120 794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:05:33.424317 794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:33.424365 794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:33.424429 794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:05:33.424474 794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:05:33.433293 794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:36975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:43407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:43407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:33.434405 794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:33.436097 794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:33.446645 800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:33.446681 802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:33.446730 799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:33.448045 794 server_base.cc:1034] running on GCE node
I20260925 15:05:33.448777 794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:33.450024 794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:33.451213 794 hybrid_clock.cc:648] HybridClock initialized: now 1790348733451164 us; error 44 us; skew 500 ppm
I20260925 15:05:33.451551 794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:05:33.453636 794 webserver.cc:460] Webserver started at http://127.22.236.62:39045/ using document root <none> and password file <none>
I20260925 15:05:33.454259 794 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:33.454348 794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:33.454643 794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:33.457329 794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "e5f9588dfaef45aba4666a9963fb13d8"
format_stamp: "Formatted at 2026-09-25 15:05:33 on dist-test-slave-nsrc"
server_key: "14f3dd7beef6695a095f3f59ca9460e1"
server_key_iv: "809980a14570d1925c621db60cbfb78c"
server_key_version: "encryptionkey@0"
I20260925 15:05:33.458045 794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "e5f9588dfaef45aba4666a9963fb13d8"
format_stamp: "Formatted at 2026-09-25 15:05:33 on dist-test-slave-nsrc"
server_key: "14f3dd7beef6695a095f3f59ca9460e1"
server_key_iv: "809980a14570d1925c621db60cbfb78c"
server_key_version: "encryptionkey@0"
I20260925 15:05:33.462718 794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:05:33.465930 808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:33.467185 794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:05:33.467332 794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "e5f9588dfaef45aba4666a9963fb13d8"
format_stamp: "Formatted at 2026-09-25 15:05:33 on dist-test-slave-nsrc"
server_key: "14f3dd7beef6695a095f3f59ca9460e1"
server_key_iv: "809980a14570d1925c621db60cbfb78c"
server_key_version: "encryptionkey@0"
I20260925 15:05:33.467522 794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:33.472757 794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:33.473425 794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:33.473682 794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:33.489290 794 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:43407
I20260925 15:05:33.489323 859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:43407 every 8 connection(s)
I20260925 15:05:33.490480 794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:05:33.493856 860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:33.499729 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 794
I20260925 15:05:33.499944 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:05:33.500751 23472 external_mini_cluster.cc:1428] Setting key 3ed9f751c4dc437023751573e0be4acb
I20260925 15:05:33.504230 860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8: Bootstrap starting.
I20260925 15:05:33.507164 860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:33.508267 860 log.cc:824] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:05:33.508765 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:33.502780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.22.236.62:43407)
0925 15:05:33.503658 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:33.503668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:33.503709 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:05:33.505385 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:33.505406 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:33.505812 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:33.506148 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:33.507134 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:33.507140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:33.507155 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:05:33.507170 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:33.507240 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:05:33.507263 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:33.507440 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:33.507574 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 15:05:33.507809 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":289,"threads_started":1}
I20260925 15:05:33.510967 860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8: No bootstrap required, opened a new log
I20260925 15:05:33.517895 860 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f9588dfaef45aba4666a9963fb13d8" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43407 } }
I20260925 15:05:33.518178 860 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:33.518245 860 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f9588dfaef45aba4666a9963fb13d8, State: Initialized, Role: FOLLOWER
I20260925 15:05:33.518728 860 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f9588dfaef45aba4666a9963fb13d8" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43407 } }
I20260925 15:05:33.518904 860 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:05:33.518992 860 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:05:33.519106 860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:33.521152 860 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f9588dfaef45aba4666a9963fb13d8" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43407 } }
I20260925 15:05:33.521610 860 leader_election.cc:302] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f9588dfaef45aba4666a9963fb13d8; no voters:
I20260925 15:05:33.522157 860 leader_election.cc:288] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:05:33.522274 865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:05:33.522838 865 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [term 1 LEADER]: Becoming Leader. State: Replica: e5f9588dfaef45aba4666a9963fb13d8, State: Running, Role: LEADER
I20260925 15:05:33.523365 865 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f9588dfaef45aba4666a9963fb13d8" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43407 } }
I20260925 15:05:33.523929 860 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:05:33.525830 866 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5f9588dfaef45aba4666a9963fb13d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f9588dfaef45aba4666a9963fb13d8" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43407 } } }
I20260925 15:05:33.525998 867 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f9588dfaef45aba4666a9963fb13d8. Latest consensus state: current_term: 1 leader_uuid: "e5f9588dfaef45aba4666a9963fb13d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f9588dfaef45aba4666a9963fb13d8" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43407 } } }
I20260925 15:05:33.526199 866 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:05:33.526320 867 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:05:33.528445 873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:05:33.532778 873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:05:33.541834 873 catalog_manager.cc:1347] Generated new cluster ID: 99a74f460b2947a4bc1b4edf6808e4b0
I20260925 15:05:33.541934 873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:05:33.572955 873 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:05:33.573338 873 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:05:33.586699 873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5f9588dfaef45aba4666a9963fb13d8: Generated new TSK 0
I20260925 15:05:33.587486 873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:05:33.603986 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43407
--builtin_ntp_servers=127.22.236.20:36975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:05:33.916246 884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:33.916518 884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:33.916646 884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:33.923636 884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:33.923736 884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:33.923877 884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:05:33.932901 884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:36975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:33.934075 884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:33.935774 884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:33.948426 889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:33.948396 892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:33.948357 890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:33.949174 884 server_base.cc:1034] running on GCE node
I20260925 15:05:33.949743 884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:33.950659 884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:33.951865 884 hybrid_clock.cc:648] HybridClock initialized: now 1790348733951812 us; error 45 us; skew 500 ppm
I20260925 15:05:33.952201 884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:05:33.954340 884 webserver.cc:460] Webserver started at http://127.22.236.1:36573/ using document root <none> and password file <none>
I20260925 15:05:33.955022 884 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:33.955117 884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:33.955416 884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:33.958236 884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "28bc862e30604420b500ae018a797d17"
format_stamp: "Formatted at 2026-09-25 15:05:33 on dist-test-slave-nsrc"
server_key: "0c5f2c74e45db6d8b120260758fb3539"
server_key_iv: "43c195d8ffba3ea5806dd8acc55573ab"
server_key_version: "encryptionkey@0"
I20260925 15:05:33.959009 884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "28bc862e30604420b500ae018a797d17"
format_stamp: "Formatted at 2026-09-25 15:05:33 on dist-test-slave-nsrc"
server_key: "0c5f2c74e45db6d8b120260758fb3539"
server_key_iv: "43c195d8ffba3ea5806dd8acc55573ab"
server_key_version: "encryptionkey@0"
I20260925 15:05:33.963883 884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:05:33.967079 898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:33.968421 884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:05:33.968569 884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "28bc862e30604420b500ae018a797d17"
format_stamp: "Formatted at 2026-09-25 15:05:33 on dist-test-slave-nsrc"
server_key: "0c5f2c74e45db6d8b120260758fb3539"
server_key_iv: "43c195d8ffba3ea5806dd8acc55573ab"
server_key_version: "encryptionkey@0"
I20260925 15:05:33.968750 884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:33.974092 884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:33.974808 884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:33.975037 884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:33.975847 884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:33.977134 884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:33.977224 884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:33.977324 884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:33.977389 884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:34.005602 884 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:39169
I20260925 15:05:34.005671 1010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:39169 every 8 connection(s)
I20260925 15:05:34.006850 884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:05:34.014477 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:34.009369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:49081 (local address 127.22.236.62:43407)
0925 15:05:34.009797 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:34.009801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:34.010826 (+ 1025us) server_negotiation.cc:407] Connection header received
0925 15:05:34.012424 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:34.012429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:34.012492 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:34.012587 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:34.013799 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:34.013804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:34.013809 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:05:34.013825 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:34.013861 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:05:34.013873 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:34.013975 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:34.014137 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 15:05:34.014196 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":204,"threads_started":1}
I20260925 15:05:34.015295 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 884
I20260925 15:05:34.015486 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:05:34.015555 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:34.009845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43407 (local address 127.22.236.1:49081)
0925 15:05:34.010687 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:34.010721 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:34.012181 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:34.012813 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:34.012821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:34.012909 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:34.013514 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:05:34.013517 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:34.013555 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:05:34.013639 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:34.013955 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:34.013965 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:34.013972 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:05:34.014218 (+ 246us) client_negotiation.cc:240] Negotiation successful
0925 15:05:34.014462 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":239,"threads_started":1}
I20260925 15:05:34.016201 23472 external_mini_cluster.cc:1428] Setting key 2675065ece779cf29b0a0c2d72d11f13
I20260925 15:05:34.016973 1011 heartbeater.cc:347] Connected to a master server at 127.22.236.62:43407
I20260925 15:05:34.017256 1011 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:34.017880 1011 heartbeater.cc:511] Master 127.22.236.62:43407 requested a full tablet report, sending...
I20260925 15:05:34.018704 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43407
--builtin_ntp_servers=127.22.236.20:36975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 15:05:34.019652 825 ts_manager.cc:194] Registered new tserver with Master: 28bc862e30604420b500ae018a797d17 (127.22.236.1:39169)
W20260925 15:05:34.326854 1015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:34.327131 1015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:34.327225 1015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:34.334432 1015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:34.334540 1015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:34.334654 1015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:05:34.343528 1015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:36975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:34.344720 1015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:34.346421 1015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:34.358302 1023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:34.358351 1020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:34.358379 1021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:34.359079 1015 server_base.cc:1034] running on GCE node
I20260925 15:05:34.359629 1015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:34.360538 1015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:34.361737 1015 hybrid_clock.cc:648] HybridClock initialized: now 1790348734361684 us; error 38 us; skew 500 ppm
I20260925 15:05:34.362042 1015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:05:34.364043 1015 webserver.cc:460] Webserver started at http://127.22.236.2:42483/ using document root <none> and password file <none>
I20260925 15:05:34.364689 1015 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:34.364786 1015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:34.365089 1015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:34.367733 1015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "5817f0b7ffea4c5eb5043645e7dfc193"
format_stamp: "Formatted at 2026-09-25 15:05:34 on dist-test-slave-nsrc"
server_key: "aa75b3a94c1286c59d875ddfc42f315c"
server_key_iv: "f42379a263dff46e5ce89d630da00e4f"
server_key_version: "encryptionkey@0"
I20260925 15:05:34.368541 1015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "5817f0b7ffea4c5eb5043645e7dfc193"
format_stamp: "Formatted at 2026-09-25 15:05:34 on dist-test-slave-nsrc"
server_key: "aa75b3a94c1286c59d875ddfc42f315c"
server_key_iv: "f42379a263dff46e5ce89d630da00e4f"
server_key_version: "encryptionkey@0"
I20260925 15:05:34.373385 1015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 15:05:34.376614 1029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:34.377911 1015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:05:34.378062 1015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "5817f0b7ffea4c5eb5043645e7dfc193"
format_stamp: "Formatted at 2026-09-25 15:05:34 on dist-test-slave-nsrc"
server_key: "aa75b3a94c1286c59d875ddfc42f315c"
server_key_iv: "f42379a263dff46e5ce89d630da00e4f"
server_key_version: "encryptionkey@0"
I20260925 15:05:34.378253 1015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:34.383626 1015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:34.384476 1015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:34.384716 1015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:34.385584 1015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:34.386909 1015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:34.387007 1015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:34.387108 1015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:34.387179 1015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:34.415572 1015 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:42613
I20260925 15:05:34.415638 1141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:42613 every 8 connection(s)
I20260925 15:05:34.416853 1015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:05:34.417472 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 1015
I20260925 15:05:34.417644 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:05:34.418166 23472 external_mini_cluster.cc:1428] Setting key 805f99836638acefb7ad77f5ee051b76
I20260925 15:05:34.420909 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43407
--builtin_ntp_servers=127.22.236.20:36975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 15:05:34.424822 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:34.419242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:46399 (local address 127.22.236.62:43407)
0925 15:05:34.419395 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:34.419400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:34.421148 (+ 1748us) server_negotiation.cc:407] Connection header received
0925 15:05:34.422767 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:34.422773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:34.422832 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:34.422921 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:34.424126 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:34.424132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:34.424155 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:05:34.424175 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:34.424212 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:05:34.424225 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:34.424352 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:34.424497 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 15:05:34.424556 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:05:34.425740 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:34.419968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43407 (local address 127.22.236.2:46399)
0925 15:05:34.421010 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:34.421054 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:34.422512 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:34.423137 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:34.423145 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:34.423235 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:34.423844 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:05:34.423846 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:34.423882 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:05:34.423963 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:34.424305 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:34.424314 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:34.424322 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:05:34.424564 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 15:05:34.424812 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":316,"threads_started":1}
I20260925 15:05:34.426976 1142 heartbeater.cc:347] Connected to a master server at 127.22.236.62:43407
I20260925 15:05:34.427261 1142 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:34.427784 1142 heartbeater.cc:511] Master 127.22.236.62:43407 requested a full tablet report, sending...
I20260925 15:05:34.429191 825 ts_manager.cc:194] Registered new tserver with Master: 5817f0b7ffea4c5eb5043645e7dfc193 (127.22.236.2:42613)
W20260925 15:05:34.727160 1145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:34.727423 1145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:34.727535 1145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:34.734578 1145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:34.734668 1145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:34.734784 1145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:05:34.743616 1145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:36975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:34.744975 1145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:34.746631 1145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:34.759163 1150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:34.759224 1153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:34.759244 1151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:34.759801 1145 server_base.cc:1034] running on GCE node
I20260925 15:05:34.760350 1145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:34.761183 1145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:34.762368 1145 hybrid_clock.cc:648] HybridClock initialized: now 1790348734762332 us; error 29 us; skew 500 ppm
I20260925 15:05:34.762650 1145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:05:34.764760 1145 webserver.cc:460] Webserver started at http://127.22.236.3:33425/ using document root <none> and password file <none>
I20260925 15:05:34.765386 1145 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:34.765475 1145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:34.765793 1145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:34.768581 1145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "4d3a44376467478496ce029cda930679"
format_stamp: "Formatted at 2026-09-25 15:05:34 on dist-test-slave-nsrc"
server_key: "3f49b4f133a1696318b509c4598dfdd7"
server_key_iv: "954cd38e975e89277c7921242bd39a41"
server_key_version: "encryptionkey@0"
I20260925 15:05:34.769356 1145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "4d3a44376467478496ce029cda930679"
format_stamp: "Formatted at 2026-09-25 15:05:34 on dist-test-slave-nsrc"
server_key: "3f49b4f133a1696318b509c4598dfdd7"
server_key_iv: "954cd38e975e89277c7921242bd39a41"
server_key_version: "encryptionkey@0"
I20260925 15:05:34.774296 1145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:05:34.777482 1159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:34.778795 1145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:05:34.778936 1145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "4d3a44376467478496ce029cda930679"
format_stamp: "Formatted at 2026-09-25 15:05:34 on dist-test-slave-nsrc"
server_key: "3f49b4f133a1696318b509c4598dfdd7"
server_key_iv: "954cd38e975e89277c7921242bd39a41"
server_key_version: "encryptionkey@0"
I20260925 15:05:34.779124 1145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:34.784494 1145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:34.785224 1145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:34.785439 1145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:34.786273 1145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:34.787554 1145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:34.787636 1145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:34.787712 1145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:34.787763 1145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:34.816025 1145 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:43409
I20260925 15:05:34.816088 1271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:43409 every 8 connection(s)
I20260925 15:05:34.817307 1145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:05:34.820283 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 1145
I20260925 15:05:34.820426 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:05:34.820837 23472 external_mini_cluster.cc:1428] Setting key 15639edb198b4349329f23ee73a7d7fd
I20260925 15:05:34.825323 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:34.819698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:59313 (local address 127.22.236.62:43407)
0925 15:05:34.819891 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:34.819897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:34.821451 (+ 1554us) server_negotiation.cc:407] Connection header received
0925 15:05:34.823330 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:34.823334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:34.823396 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:34.823475 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:34.824742 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:34.824746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:34.824749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:05:34.824763 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:34.824792 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:05:34.824802 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:34.824928 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:34.825080 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 15:05:34.825126 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:05:34.826232 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:34.820272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43407 (local address 127.22.236.3:59313)
0925 15:05:34.821295 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:34.821340 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:34.823085 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:34.823707 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:34.823715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:34.823847 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:34.824473 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:05:34.824475 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:34.824513 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:05:34.824602 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:34.824889 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:34.824898 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:34.824916 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 15:05:34.825142 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 15:05:34.825347 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":315,"threads_started":1}
I20260925 15:05:34.827342 1272 heartbeater.cc:347] Connected to a master server at 127.22.236.62:43407
I20260925 15:05:34.827589 1272 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:34.828153 1272 heartbeater.cc:511] Master 127.22.236.62:43407 requested a full tablet report, sending...
I20260925 15:05:34.829329 825 ts_manager.cc:194] Registered new tserver with Master: 4d3a44376467478496ce029cda930679 (127.22.236.3:43409)
I20260925 15:05:34.835456 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:05:34.841038 1013 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:05:34.839604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.22.236.62:43407)
0925 15:05:34.839767 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:34.839771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:34.840242 (+ 471us) server_negotiation.cc:407] Connection header received
0925 15:05:34.840369 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:34.840372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:34.840419 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:34.840490 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:34.840787 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46250: BlockingRecv error: recv got EOF from 127.0.0.1:46250 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:05:34.841506 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 884
I20260925 15:05:34.844755 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:34.842415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48578 (local address 127.22.236.1:39169)
0925 15:05:34.842852 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:34.842859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:34.842881 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:05:34.842940 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:34.842945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:34.843139 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:34.843245 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:34.844110 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:34.844116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:34.844161 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:05:34.844185 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:34.844228 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:05:34.844244 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:34.844338 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:34.844427 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 15:05:34.844503 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":225,"threads_started":1}
I20260925 15:05:34.845834 1006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:34.923878 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 884
I20260925 15:05:34.937213 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 1015
I20260925 15:05:34.940445 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:34.938153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.22.236.2:42613)
0925 15:05:34.938640 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:34.938646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:34.938672 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:05:34.938733 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:34.938739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:34.938941 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:34.939063 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:34.939866 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:34.939872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:34.939884 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:05:34.939899 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:34.939936 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:05:34.939949 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:34.940046 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:34.940097 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 15:05:34.940218 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260925 15:05:34.941609 1137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:35.019167 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 1015
I20260925 15:05:35.031069 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 1145
I20260925 15:05:35.034368 1285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:35.031890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.22.236.3:43409)
0925 15:05:35.032448 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:35.032455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:35.032485 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:05:35.032544 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:35.032550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:35.032767 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:35.032882 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:35.033797 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:35.033803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:35.033814 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:05:35.033825 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:35.033863 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:05:35.033879 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:35.034000 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:35.034043 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 15:05:35.034122 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":279,"threads_started":1}
I20260925 15:05:35.035495 1267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:35.115283 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 1145
I20260925 15:05:35.127285 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 794
I20260925 15:05:35.128449 855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:35.225131 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 794
2026-09-25T15:05:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2174 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:05:35 dist-test-slave-nsrc krb5kdc[1290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:05:35 dist-test-slave-nsrc krb5kdc[1290](info): set up 2 sockets
Sep 25 15:05:35 dist-test-slave-nsrc krb5kdc[1290](info): commencing operation
krb5kdc: starting...
W20260925 15:05:36.233031 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:05:36 dist-test-slave-nsrc krb5kdc[1290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:05:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:05:36.445297 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:41009
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:44057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:41009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:05:36.754735 1303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:36.755021 1303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:36.755141 1303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:36.762259 1303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:05:36.762364 1303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:36.762421 1303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:36.762502 1303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:05:36.762554 1303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:05:36.772188 1303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:44057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:41009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:41009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:36.773524 1303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:36.774719 1303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:36.785482 1311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:36.785517 1309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:36.785534 1308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:36.786505 1303 server_base.cc:1034] running on GCE node
I20260925 15:05:36.787278 1303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:36.788734 1303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:36.790042 1303 hybrid_clock.cc:648] HybridClock initialized: now 1790348736789916 us; error 36 us; skew 500 ppm
I20260925 15:05:36.790455 1303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:36 dist-test-slave-nsrc krb5kdc[1290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348736, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:36.799132 1303 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:05:36.800649 1303 webserver.cc:460] Webserver started at http://127.22.236.62:35303/ using document root <none> and password file <none>
I20260925 15:05:36.801317 1303 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:36.801426 1303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:36.801714 1303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:36.804682 1303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "d5ff295611a04e3d9a1bdd5aaba3b4cd"
format_stamp: "Formatted at 2026-09-25 15:05:36 on dist-test-slave-nsrc"
server_key: "382b58eab77ec724fa4e7c6b838cf090"
server_key_iv: "de14cd2d4d41222cb8733af387cfe1bd"
server_key_version: "encryptionkey@0"
I20260925 15:05:36.805410 1303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "d5ff295611a04e3d9a1bdd5aaba3b4cd"
format_stamp: "Formatted at 2026-09-25 15:05:36 on dist-test-slave-nsrc"
server_key: "382b58eab77ec724fa4e7c6b838cf090"
server_key_iv: "de14cd2d4d41222cb8733af387cfe1bd"
server_key_version: "encryptionkey@0"
I20260925 15:05:36.810262 1303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:05:36.813495 1318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:36.814740 1303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 15:05:36.814891 1303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "d5ff295611a04e3d9a1bdd5aaba3b4cd"
format_stamp: "Formatted at 2026-09-25 15:05:36 on dist-test-slave-nsrc"
server_key: "382b58eab77ec724fa4e7c6b838cf090"
server_key_iv: "de14cd2d4d41222cb8733af387cfe1bd"
server_key_version: "encryptionkey@0"
I20260925 15:05:36.815063 1303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:36.831835 1303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:36.835690 1303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:36.835935 1303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:36.852208 1303 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:41009
I20260925 15:05:36.852229 1369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:41009 every 8 connection(s)
I20260925 15:05:36.853461 1303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:05:36.854825 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 1303
I20260925 15:05:36.855017 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:05:36.855600 23472 external_mini_cluster.cc:1428] Setting key 120172c09d54ed0ed0645641a9a6daba
I20260925 15:05:36.857342 1370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:36.868043 1370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd: Bootstrap starting.
Sep 25 15:05:36 dist-test-slave-nsrc krb5kdc[1290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:05:36.871460 1370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:36.872483 1370 log.cc:824] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:05:36.875255 1370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd: No bootstrap required, opened a new log
I20260925 15:05:36.876006 1373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:36.857434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38106 (local address 127.22.236.62:41009)
0925 15:05:36.858188 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:36.858198 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:36.858237 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:05:36.859734 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:36.859754 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:36.860129 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:36.860540 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:36.861933 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:36.863200 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:36.864470 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:36.865132 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:36.865293 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:36.869186 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:36.869194 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:36.869208 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:36.869246 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:36.872179 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:36.872840 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:36.872849 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:36.872858 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:36.873064 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:36.873397 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:36.873402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:36.873406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:36.873913 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:36.874195 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:36.874591 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 15:05:36.874981 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":306,"threads_started":1}
I20260925 15:05:36.882682 1370 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ff295611a04e3d9a1bdd5aaba3b4cd" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 41009 } }
I20260925 15:05:36.883008 1370 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:36.883091 1370 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ff295611a04e3d9a1bdd5aaba3b4cd, State: Initialized, Role: FOLLOWER
I20260925 15:05:36.883574 1370 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ff295611a04e3d9a1bdd5aaba3b4cd" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 41009 } }
I20260925 15:05:36.883769 1370 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:05:36.883879 1370 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:05:36.884032 1370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:36.886087 1370 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ff295611a04e3d9a1bdd5aaba3b4cd" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 41009 } }
I20260925 15:05:36.886574 1370 leader_election.cc:302] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5ff295611a04e3d9a1bdd5aaba3b4cd; no voters:
I20260925 15:05:36.887101 1370 leader_election.cc:288] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:05:36.887202 1375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:05:36.887800 1375 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [term 1 LEADER]: Becoming Leader. State: Replica: d5ff295611a04e3d9a1bdd5aaba3b4cd, State: Running, Role: LEADER
I20260925 15:05:36.888363 1375 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ff295611a04e3d9a1bdd5aaba3b4cd" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 41009 } }
I20260925 15:05:36.888916 1370 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:05:36.890681 1376 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5ff295611a04e3d9a1bdd5aaba3b4cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ff295611a04e3d9a1bdd5aaba3b4cd" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 41009 } } }
I20260925 15:05:36.891024 1376 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:05:36.891793 1380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:05:36.891605 1377 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ff295611a04e3d9a1bdd5aaba3b4cd. Latest consensus state: current_term: 1 leader_uuid: "d5ff295611a04e3d9a1bdd5aaba3b4cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ff295611a04e3d9a1bdd5aaba3b4cd" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 41009 } } }
I20260925 15:05:36.891934 1377 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:05:36.896775 1380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:05:36.904109 1380 catalog_manager.cc:1347] Generated new cluster ID: 99bde3ae9440483b9e92e4f7fb36a6ce
I20260925 15:05:36.904232 1380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:05:36.925314 1380 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:05:36.927176 1380 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:05:36.935868 1380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd: Generated new TSK 0
I20260925 15:05:36.936874 1380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:05:36.985745 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:41009
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:44057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:05:37.299556 1396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:37.299824 1396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:37.299912 1396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:37.307060 1396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:37.307173 1396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:37.307292 1396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:05:37.316926 1396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:44057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:41009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:37.318159 1396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:37.319442 1396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:37.331560 1401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:37.331524 1402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:37.331722 1404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:37.332672 1396 server_base.cc:1034] running on GCE node
I20260925 15:05:37.333257 1396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:37.334142 1396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:37.335330 1396 hybrid_clock.cc:648] HybridClock initialized: now 1790348737335275 us; error 47 us; skew 500 ppm
I20260925 15:05:37.335628 1396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:37 dist-test-slave-nsrc krb5kdc[1290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348737, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:37.338593 1396 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:05:37.339898 1396 webserver.cc:460] Webserver started at http://127.22.236.1:41993/ using document root <none> and password file <none>
I20260925 15:05:37.340610 1396 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:37.340720 1396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:37.340986 1396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:37.343665 1396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "3ccab495427047a296668ea3434ff0d1"
format_stamp: "Formatted at 2026-09-25 15:05:37 on dist-test-slave-nsrc"
server_key: "e0f48e3a1d7403ac84d0744e60ce6d6d"
server_key_iv: "3d919044611c731ac8bc4d1640531794"
server_key_version: "encryptionkey@0"
I20260925 15:05:37.344419 1396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "3ccab495427047a296668ea3434ff0d1"
format_stamp: "Formatted at 2026-09-25 15:05:37 on dist-test-slave-nsrc"
server_key: "e0f48e3a1d7403ac84d0744e60ce6d6d"
server_key_iv: "3d919044611c731ac8bc4d1640531794"
server_key_version: "encryptionkey@0"
I20260925 15:05:37.349242 1396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:05:37.352540 1411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:37.353899 1396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:05:37.354086 1396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "3ccab495427047a296668ea3434ff0d1"
format_stamp: "Formatted at 2026-09-25 15:05:37 on dist-test-slave-nsrc"
server_key: "e0f48e3a1d7403ac84d0744e60ce6d6d"
server_key_iv: "3d919044611c731ac8bc4d1640531794"
server_key_version: "encryptionkey@0"
I20260925 15:05:37.354264 1396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:37.386881 1396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:37.391095 1396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:37.391377 1396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:37.392364 1396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:37.393651 1396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:37.393724 1396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:37.393843 1396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:37.393893 1396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:37.422073 1396 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:44455
I20260925 15:05:37.422106 1523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:44455 every 8 connection(s)
I20260925 15:05:37.423261 1396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:05:37.425719 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 1396
I20260925 15:05:37.425930 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:05:37.426519 23472 external_mini_cluster.cc:1428] Setting key cadea410375e2986aefa5e644ae44747
Sep 25 15:05:37 dist-test-slave-nsrc krb5kdc[1290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348737, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:05:37.442648 1526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:37.425616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:41901 (local address 127.22.236.62:41009)
0925 15:05:37.426074 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:37.426080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:37.427072 (+ 992us) server_negotiation.cc:407] Connection header received
0925 15:05:37.429162 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:37.429167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:37.429242 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:37.429348 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:37.430850 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:37.431785 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:37.433331 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:37.433860 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:37.433970 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:37.437591 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:37.437596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:37.437599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:37.437632 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:37.439857 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:37.440543 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:37.440548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:37.440550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:37.440659 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:37.441039 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:37.441043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:37.441045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:37.441311 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:37.441430 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:37.442075 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 15:05:37.442302 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":222,"threads_started":1}
I20260925 15:05:37.443606 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:37.426165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:41009 (local address 127.22.236.1:41901)
0925 15:05:37.426899 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:37.426951 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:37.428886 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:37.429578 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:37.429586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:37.430096 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:37.430689 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:37.430708 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:37.431930 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:37.431933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:37.433207 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:37.433218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:37.434092 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:37.434097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:37.434296 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:37.435014 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:37.435044 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:37.437388 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:37.439977 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:37.439986 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:37.440002 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:37.440430 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:37.440755 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:37.440760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:37.440764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:37.440934 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:37.441423 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:37.441430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:37.441748 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 15:05:37.441979 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 15:05:37.442467 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":308,"threads_started":1}
I20260925 15:05:37.445080 1524 heartbeater.cc:347] Connected to a master server at 127.22.236.62:41009
I20260925 15:05:37.445353 1524 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:37.446010 1524 heartbeater.cc:511] Master 127.22.236.62:41009 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
I20260925 15:05:37.447887 1335 ts_manager.cc:194] Registered new tserver with Master: 3ccab495427047a296668ea3434ff0d1 (127.22.236.1:44455)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
I20260925 15:05:37.449383 1335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:41901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:05:37.471271 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:41009
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:44057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:05:37.778957 1530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:37.779228 1530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:37.779325 1530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:37.786238 1530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:37.786338 1530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:37.786454 1530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:05:37.795970 1530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:44057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:41009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:37.797233 1530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:37.798477 1530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:37.810423 1535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:37.810650 1536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:37.810683 1538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:37.811160 1530 server_base.cc:1034] running on GCE node
I20260925 15:05:37.811729 1530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:37.812634 1530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:37.813817 1530 hybrid_clock.cc:648] HybridClock initialized: now 1790348737813774 us; error 33 us; skew 500 ppm
I20260925 15:05:37.814080 1530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:37 dist-test-slave-nsrc krb5kdc[1290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348737, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:37.816879 1530 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:05:37.818226 1530 webserver.cc:460] Webserver started at http://127.22.236.2:33495/ using document root <none> and password file <none>
I20260925 15:05:37.818859 1530 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:37.818982 1530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:37.819242 1530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:37.821941 1530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "f047583bf76440f1b883c8c98d25af4b"
format_stamp: "Formatted at 2026-09-25 15:05:37 on dist-test-slave-nsrc"
server_key: "0a326f55779872677088bb6baace0976"
server_key_iv: "21c163232a93d17d860d22154a3bc2f4"
server_key_version: "encryptionkey@0"
I20260925 15:05:37.822623 1530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "f047583bf76440f1b883c8c98d25af4b"
format_stamp: "Formatted at 2026-09-25 15:05:37 on dist-test-slave-nsrc"
server_key: "0a326f55779872677088bb6baace0976"
server_key_iv: "21c163232a93d17d860d22154a3bc2f4"
server_key_version: "encryptionkey@0"
I20260925 15:05:37.827276 1530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 15:05:37.830492 1545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:37.831772 1530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:05:37.831909 1530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "f047583bf76440f1b883c8c98d25af4b"
format_stamp: "Formatted at 2026-09-25 15:05:37 on dist-test-slave-nsrc"
server_key: "0a326f55779872677088bb6baace0976"
server_key_iv: "21c163232a93d17d860d22154a3bc2f4"
server_key_version: "encryptionkey@0"
I20260925 15:05:37.832078 1530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:37.856468 1530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:37.860468 1530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:37.860734 1530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:37.861603 1530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:37.862810 1530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:37.862881 1530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:37.862993 1530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:37.863041 1530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:37.891517 1530 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:38333
I20260925 15:05:37.891580 1657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:38333 every 8 connection(s)
I20260925 15:05:37.892767 1530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:05:37.900355 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 1530
I20260925 15:05:37.900533 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:05:37.901070 23472 external_mini_cluster.cc:1428] Setting key 2018457f5db2584d5aa2914180e4235c
Sep 25 15:05:37 dist-test-slave-nsrc krb5kdc[1290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348737, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:05:37.912369 1526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:37.895113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:59543 (local address 127.22.236.62:41009)
0925 15:05:37.895278 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:37.895282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:37.896392 (+ 1110us) server_negotiation.cc:407] Connection header received
0925 15:05:37.898185 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:37.898189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:37.898253 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:37.898335 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:37.899678 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:37.900458 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:37.902416 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:37.902998 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:37.903099 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:37.906900 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:37.906905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:37.906908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:37.906938 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:37.909276 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:37.910084 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:37.910089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:37.910092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:37.910245 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:37.910651 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:37.910656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:37.910660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:37.910994 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:37.911120 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:37.911800 (+ 680us) server_negotiation.cc:299] Negotiation successful
0925 15:05:37.912021 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:05:37.913275 1660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:37.895544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:41009 (local address 127.22.236.2:59543)
0925 15:05:37.896245 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:37.896283 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:37.897953 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:37.898510 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:37.898517 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:37.898992 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:37.899523 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:37.899539 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:37.900613 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:37.900619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:37.902288 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:37.902299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:37.903225 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:37.903232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:37.903497 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:37.904184 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:37.904214 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:37.906697 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:37.909406 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:37.909415 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:37.909435 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:37.909976 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:37.910358 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:37.910361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:37.910364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:37.910549 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:37.911121 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:37.911131 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:37.911456 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 15:05:37.911692 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 15:05:37.912101 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":242,"threads_started":1}
I20260925 15:05:37.914599 1658 heartbeater.cc:347] Connected to a master server at 127.22.236.62:41009
I20260925 15:05:37.914870 1658 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:37.915535 1658 heartbeater.cc:511] Master 127.22.236.62:41009 requested a full tablet report, sending...
I20260925 15:05:37.916900 1335 ts_manager.cc:194] Registered new tserver with Master: f047583bf76440f1b883c8c98d25af4b (127.22.236.2:38333)
I20260925 15:05:37.917666 1335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:59543
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:05:37.946786 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:41009
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:44057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:05:38.257437 1663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:05:38.257721 1663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:05:38.257815 1663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:05:38.264937 1663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:05:38.265064 1663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:05:38.265184 1663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:05:38.274814 1663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:44057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:41009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:05:38.276134 1663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:05:38.277401 1663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:05:38.289475 1668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:38.289409 1669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:05:38.289444 1671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:05:38.290427 1663 server_base.cc:1034] running on GCE node
I20260925 15:05:38.290972 1663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:05:38.291822 1663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:05:38.293044 1663 hybrid_clock.cc:648] HybridClock initialized: now 1790348738292989 us; error 41 us; skew 500 ppm
I20260925 15:05:38.293390 1663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:05:38 dist-test-slave-nsrc krb5kdc[1290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348738, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:05:38.296439 1663 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:05:38.297827 1663 webserver.cc:460] Webserver started at http://127.22.236.3:33863/ using document root <none> and password file <none>
I20260925 15:05:38.298476 1663 fs_manager.cc:359] Metadata directory not provided
I20260925 15:05:38.298586 1663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:05:38.298848 1663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:05:38.301797 1663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "5df334d031b047a2b55da943a2fd37bb"
format_stamp: "Formatted at 2026-09-25 15:05:38 on dist-test-slave-nsrc"
server_key: "47b05535c1d30066e031bb2bafa65c94"
server_key_iv: "80c13834a465b4495a227afac5a150ee"
server_key_version: "encryptionkey@0"
I20260925 15:05:38.302500 1663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "5df334d031b047a2b55da943a2fd37bb"
format_stamp: "Formatted at 2026-09-25 15:05:38 on dist-test-slave-nsrc"
server_key: "47b05535c1d30066e031bb2bafa65c94"
server_key_iv: "80c13834a465b4495a227afac5a150ee"
server_key_version: "encryptionkey@0"
I20260925 15:05:38.307365 1663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 15:05:38.310631 1678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:38.311959 1663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:05:38.312183 1663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "5df334d031b047a2b55da943a2fd37bb"
format_stamp: "Formatted at 2026-09-25 15:05:38 on dist-test-slave-nsrc"
server_key: "47b05535c1d30066e031bb2bafa65c94"
server_key_iv: "80c13834a465b4495a227afac5a150ee"
server_key_version: "encryptionkey@0"
I20260925 15:05:38.312367 1663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:05:38.337939 1663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:05:38.341737 1663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:05:38.341997 1663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:05:38.342891 1663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:05:38.344107 1663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:05:38.344213 1663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:38.344326 1663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:05:38.344378 1663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:05:38.372954 1663 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:44449
I20260925 15:05:38.373000 1790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:44449 every 8 connection(s)
I20260925 15:05:38.374162 1663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:05:38.380507 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 1663
I20260925 15:05:38.380671 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:05:38.381080 23472 external_mini_cluster.cc:1428] Setting key 6d9a7f1febf92a4cca1b9101858c76be
Sep 25 15:05:38 dist-test-slave-nsrc krb5kdc[1290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348738, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:05:38.392757 1526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.376524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:54017 (local address 127.22.236.62:41009)
0925 15:05:38.376697 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:38.376702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:38.377842 (+ 1140us) server_negotiation.cc:407] Connection header received
0925 15:05:38.379557 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:38.379563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:38.379641 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:38.379723 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:38.381250 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.382074 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.383680 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.384061 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.384190 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.387318 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:38.387323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:38.387326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:38.387357 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:38.389823 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:38.390502 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:38.390506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:38.390509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:38.390623 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:38.391022 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:38.391027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:38.391030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:38.391310 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:38.391434 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:38.392197 (+ 763us) server_negotiation.cc:299] Negotiation successful
0925 15:05:38.392418 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:05:38.393801 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.377036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:41009 (local address 127.22.236.3:54017)
0925 15:05:38.377697 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:38.377733 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:38.379305 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:38.379951 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:38.379959 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:38.380498 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:05:38.381094 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:38.381111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.382202 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.382208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:38.383547 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:38.383558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.384283 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.384290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:38.384507 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.385114 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:05:38.385138 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:05:38.387129 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:05:38.389956 (+ 2827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:38.389966 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:38.389985 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:38.390393 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:38.390723 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:05:38.390728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:05:38.390732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:05:38.390921 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:05:38.391434 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:05:38.391443 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:05:38.391801 (+ 358us) client_negotiation.cc:769] Sending connection context
0925 15:05:38.392055 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 15:05:38.392618 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":240,"threads_started":1}
I20260925 15:05:38.395195 1791 heartbeater.cc:347] Connected to a master server at 127.22.236.62:41009
I20260925 15:05:38.395506 1791 heartbeater.cc:464] Registering TS with master...
I20260925 15:05:38.396342 1791 heartbeater.cc:511] Master 127.22.236.62:41009 requested a full tablet report, sending...
I20260925 15:05:38.397617 1335 ts_manager.cc:194] Registered new tserver with Master: 5df334d031b047a2b55da943a2fd37bb (127.22.236.3:44449)
I20260925 15:05:38.398343 1335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:54017
I20260925 15:05:38.407927 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:05:38.423314 1526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.412031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38118 (local address 127.22.236.62:41009)
0925 15:05:38.412237 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:38.412244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:38.412592 (+ 348us) server_negotiation.cc:407] Connection header received
0925 15:05:38.412716 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:38.412719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:38.412775 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:38.412848 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:38.414188 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.414870 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.416014 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.416435 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.416524 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.418573 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:38.418577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:38.418580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:38.418610 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:38.420965 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:38.421521 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:38.421525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:38.421527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:38.421680 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:38.422015 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:38.422020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:38.422023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:38.422318 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:38.422530 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:38.422853 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 15:05:38.423047 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:05:38.427199 1335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:05:38.433631 1335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:05:38.452387 1524 heartbeater.cc:503] Master 127.22.236.62:41009 was elected leader, sending a full tablet report...
I20260925 15:05:38.460919 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.444746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:44449 (local address 127.0.0.1:35236)
0925 15:05:38.451226 (+ 6480us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:38.451245 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:38.451344 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:38.452597 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:38.452606 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:38.452639 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:38.453066 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:38.453078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.455423 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.455429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:38.458062 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:38.458078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.459946 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.459953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:38.460189 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.460243 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 15:05:38.460353 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:05:38.460535 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6363,"thread_start_us":6313,"threads_started":1}
I20260925 15:05:38.461473 1803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.444582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37576 (local address 127.22.236.2:38333)
0925 15:05:38.452888 (+ 8306us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:38.452896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:38.452924 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:05:38.453012 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:38.453019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:38.453246 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:38.453429 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:38.454279 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.455896 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.458523 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.459951 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.460070 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.460284 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:38.461019 (+ 735us) server_negotiation.cc:299] Negotiation successful
0925 15:05:38.461178 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7453,"thread_start_us":7329,"threads_started":1}
I20260925 15:05:38.461463 1806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.445253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.22.236.3:44449)
0925 15:05:38.451872 (+ 6619us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:38.451880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:38.451915 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:05:38.452024 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:38.452033 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:38.452343 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:38.453275 (+ 932us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:38.453542 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.455257 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.458285 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.459750 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.460725 (+ 975us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.460950 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:38.461017 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 15:05:38.461183 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6460,"thread_start_us":6312,"threads_started":1}
I20260925 15:05:38.462008 1804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.444616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:38333 (local address 127.0.0.1:37576)
0925 15:05:38.450660 (+ 6044us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:38.450697 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:38.450875 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:38.453670 (+ 2795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:38.453675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:38.453697 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:38.454062 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:38.454076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.456061 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.456067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:38.458334 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:38.458351 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.460415 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.460421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:38.460584 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.460613 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:05:38.460705 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:05:38.460830 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5790,"thread_start_us":5670,"threads_started":1}
I20260925 15:05:38.465226 1593 tablet_service.cc:1467] Processing CreateTablet for tablet 49ec0b2fdcc44fce95a2d3c932a7f776 (DEFAULT_TABLE table=test-table [id=067ac443c81547ecbf06ccba0d0c9a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:38.465201 1726 tablet_service.cc:1467] Processing CreateTablet for tablet 49ec0b2fdcc44fce95a2d3c932a7f776 (DEFAULT_TABLE table=test-table [id=067ac443c81547ecbf06ccba0d0c9a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:38.466589 1593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ec0b2fdcc44fce95a2d3c932a7f776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:38.466588 1726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ec0b2fdcc44fce95a2d3c932a7f776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:38.466858 1808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.452243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:44455 (local address 127.0.0.1:56746)
0925 15:05:38.459891 (+ 7648us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:38.459909 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:38.460076 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:38.460880 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:38.460885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:38.460913 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:38.461563 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:38.461575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.463279 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.463284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:38.464766 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:38.464778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.466193 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.466197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:38.466332 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.466361 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:05:38.466443 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:05:38.466555 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7512,"thread_start_us":7382,"threads_started":1}
I20260925 15:05:38.467497 1807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.451804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.22.236.1:44455)
0925 15:05:38.459599 (+ 7795us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:38.459607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:38.460213 (+ 606us) server_negotiation.cc:407] Connection header received
0925 15:05:38.460327 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:38.460336 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:38.460684 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:38.461078 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:38.461728 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.463144 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.464902 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.465782 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.465941 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.466104 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:38.467052 (+ 948us) server_negotiation.cc:299] Negotiation successful
0925 15:05:38.467210 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7640,"thread_start_us":7510,"threads_started":1}
I20260925 15:05:38.470690 1459 tablet_service.cc:1467] Processing CreateTablet for tablet 49ec0b2fdcc44fce95a2d3c932a7f776 (DEFAULT_TABLE table=test-table [id=067ac443c81547ecbf06ccba0d0c9a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:05:38.472111 1459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ec0b2fdcc44fce95a2d3c932a7f776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:05:38.480360 1809 tablet_bootstrap.cc:492] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b: Bootstrap starting.
I20260925 15:05:38.482290 1810 tablet_bootstrap.cc:492] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb: Bootstrap starting.
I20260925 15:05:38.485028 1809 tablet_bootstrap.cc:654] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:38.485349 1811 tablet_bootstrap.cc:492] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1: Bootstrap starting.
I20260925 15:05:38.485739 1810 tablet_bootstrap.cc:654] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:38.486433 1809 log.cc:824] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:05:38.486909 1810 log.cc:824] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb: Log is configured to *not* fsync() on all Append() calls
I20260925 15:05:38.488076 1811 tablet_bootstrap.cc:654] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1: Neither blocks nor log segments found. Creating new log.
I20260925 15:05:38.489283 1811 log.cc:824] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:05:38.489482 1809 tablet_bootstrap.cc:492] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b: No bootstrap required, opened a new log
I20260925 15:05:38.489702 1810 tablet_bootstrap.cc:492] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb: No bootstrap required, opened a new log
I20260925 15:05:38.489714 1809 ts_tablet_manager.cc:1397] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260925 15:05:38.489959 1810 ts_tablet_manager.cc:1397] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 15:05:38.492089 1811 tablet_bootstrap.cc:492] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1: No bootstrap required, opened a new log
I20260925 15:05:38.492343 1811 ts_tablet_manager.cc:1397] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260925 15:05:38.497442 1809 raft_consensus.cc:348] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f047583bf76440f1b883c8c98d25af4b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38333 } } peers { permanent_uuid: "5df334d031b047a2b55da943a2fd37bb" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44449 } } peers { permanent_uuid: "3ccab495427047a296668ea3434ff0d1" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44455 } }
I20260925 15:05:38.497916 1809 raft_consensus.cc:374] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:38.498032 1809 raft_consensus.cc:729] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f047583bf76440f1b883c8c98d25af4b, State: Initialized, Role: FOLLOWER
I20260925 15:05:38.498620 1809 consensus_queue.cc:260] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f047583bf76440f1b883c8c98d25af4b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38333 } } peers { permanent_uuid: "5df334d031b047a2b55da943a2fd37bb" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44449 } } peers { permanent_uuid: "3ccab495427047a296668ea3434ff0d1" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44455 } }
I20260925 15:05:38.499615 1658 heartbeater.cc:503] Master 127.22.236.62:41009 was elected leader, sending a full tablet report...
I20260925 15:05:38.499771 1810 raft_consensus.cc:348] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f047583bf76440f1b883c8c98d25af4b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38333 } } peers { permanent_uuid: "5df334d031b047a2b55da943a2fd37bb" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44449 } } peers { permanent_uuid: "3ccab495427047a296668ea3434ff0d1" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44455 } }
I20260925 15:05:38.500172 1810 raft_consensus.cc:374] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:38.500259 1810 raft_consensus.cc:729] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df334d031b047a2b55da943a2fd37bb, State: Initialized, Role: FOLLOWER
I20260925 15:05:38.500308 1809 ts_tablet_manager.cc:1428] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b: Time spent starting tablet: real 0.010s user 0.001s sys 0.009s
I20260925 15:05:38.500860 1810 consensus_queue.cc:260] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f047583bf76440f1b883c8c98d25af4b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38333 } } peers { permanent_uuid: "5df334d031b047a2b55da943a2fd37bb" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44449 } } peers { permanent_uuid: "3ccab495427047a296668ea3434ff0d1" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44455 } }
I20260925 15:05:38.501873 1791 heartbeater.cc:503] Master 127.22.236.62:41009 was elected leader, sending a full tablet report...
I20260925 15:05:38.502151 1811 raft_consensus.cc:348] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f047583bf76440f1b883c8c98d25af4b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38333 } } peers { permanent_uuid: "5df334d031b047a2b55da943a2fd37bb" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44449 } } peers { permanent_uuid: "3ccab495427047a296668ea3434ff0d1" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44455 } }
I20260925 15:05:38.502453 1810 ts_tablet_manager.cc:1428] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260925 15:05:38.502528 1811 raft_consensus.cc:374] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:05:38.502604 1811 raft_consensus.cc:729] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccab495427047a296668ea3434ff0d1, State: Initialized, Role: FOLLOWER
I20260925 15:05:38.503152 1811 consensus_queue.cc:260] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f047583bf76440f1b883c8c98d25af4b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38333 } } peers { permanent_uuid: "5df334d031b047a2b55da943a2fd37bb" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44449 } } peers { permanent_uuid: "3ccab495427047a296668ea3434ff0d1" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44455 } }
I20260925 15:05:38.504671 1811 ts_tablet_manager.cc:1428] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
W20260925 15:05:38.625700 1792 tablet.cc:2369] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:05:38.641392 1817 raft_consensus.cc:482] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:05:38.641654 1817 raft_consensus.cc:504] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f047583bf76440f1b883c8c98d25af4b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38333 } } peers { permanent_uuid: "5df334d031b047a2b55da943a2fd37bb" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44449 } } peers { permanent_uuid: "3ccab495427047a296668ea3434ff0d1" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44455 } }
I20260925 15:05:38.643286 1817 leader_election.cc:288] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f047583bf76440f1b883c8c98d25af4b (127.22.236.2:38333), 5df334d031b047a2b55da943a2fd37bb (127.22.236.3:44449)
W20260925 15:05:38.644582 1659 tablet.cc:2369] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:05:38.650627 1806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.643820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:41535 (local address 127.22.236.3:44449)
0925 15:05:38.644000 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:38.644005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:38.645155 (+ 1150us) server_negotiation.cc:407] Connection header received
0925 15:05:38.645316 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:38.645323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:38.645398 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:38.645477 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:38.646036 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.647050 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.649047 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.649777 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.649911 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.649981 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:38.650271 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 15:05:38.650387 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:05:38.650621 1803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.643753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:39491 (local address 127.22.236.2:38333)
0925 15:05:38.643944 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:38.643950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:38.644759 (+ 809us) server_negotiation.cc:407] Connection header received
0925 15:05:38.645055 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:38.645061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:38.645138 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:38.645221 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:05:38.645823 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.646928 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.648999 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.649737 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.649866 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.649936 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:38.650270 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 15:05:38.650388 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:05:38.650774 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.643647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:44449 (local address 127.22.236.1:41535)
0925 15:05:38.645045 (+ 1398us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:38.645062 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:38.645190 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:38.645512 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:38.645516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:38.645535 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:38.645903 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:38.645912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.647174 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.647178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:38.648894 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:38.648906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.649988 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.649995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:38.650131 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.650158 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 15:05:38.650235 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:05:38.650353 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1280,"mutex_wait_us":29,"thread_start_us":487,"threads_started":1}
I20260925 15:05:38.650799 1818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.643577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:38333 (local address 127.22.236.1:39491)
0925 15:05:38.644628 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0925 15:05:38.644654 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:05:38.644814 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:05:38.645324 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:05:38.645328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:05:38.645351 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:05:38.645690 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:38.645702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.647056 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.647061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:38.648851 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:05:38.648865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.649946 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.649953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:05:38.650091 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.650149 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 15:05:38.650227 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:05:38.650346 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":86,"thread_start_us":708,"threads_started":1}
I20260925 15:05:38.651741 1746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ec0b2fdcc44fce95a2d3c932a7f776" candidate_uuid: "3ccab495427047a296668ea3434ff0d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df334d031b047a2b55da943a2fd37bb" is_pre_election: true
I20260925 15:05:38.651738 1613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ec0b2fdcc44fce95a2d3c932a7f776" candidate_uuid: "3ccab495427047a296668ea3434ff0d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f047583bf76440f1b883c8c98d25af4b" is_pre_election: true
I20260925 15:05:38.652185 1746 raft_consensus.cc:2454] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ccab495427047a296668ea3434ff0d1 in term 0.
I20260925 15:05:38.652192 1613 raft_consensus.cc:2454] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ccab495427047a296668ea3434ff0d1 in term 0.
I20260925 15:05:38.652745 1414 leader_election.cc:302] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccab495427047a296668ea3434ff0d1, 5df334d031b047a2b55da943a2fd37bb; no voters:
I20260925 15:05:38.653069 1817 raft_consensus.cc:2781] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:05:38.653201 1817 raft_consensus.cc:482] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:05:38.653306 1817 raft_consensus.cc:3037] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:38.655287 1817 raft_consensus.cc:504] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f047583bf76440f1b883c8c98d25af4b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38333 } } peers { permanent_uuid: "5df334d031b047a2b55da943a2fd37bb" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44449 } } peers { permanent_uuid: "3ccab495427047a296668ea3434ff0d1" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44455 } }
I20260925 15:05:38.655786 1817 leader_election.cc:288] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [CANDIDATE]: Term 1 election: Requested vote from peers f047583bf76440f1b883c8c98d25af4b (127.22.236.2:38333), 5df334d031b047a2b55da943a2fd37bb (127.22.236.3:44449)
I20260925 15:05:38.656260 1746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ec0b2fdcc44fce95a2d3c932a7f776" candidate_uuid: "3ccab495427047a296668ea3434ff0d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df334d031b047a2b55da943a2fd37bb"
I20260925 15:05:38.656260 1613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ec0b2fdcc44fce95a2d3c932a7f776" candidate_uuid: "3ccab495427047a296668ea3434ff0d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f047583bf76440f1b883c8c98d25af4b"
I20260925 15:05:38.656450 1746 raft_consensus.cc:3037] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:38.656450 1613 raft_consensus.cc:3037] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:05:38.659412 1613 raft_consensus.cc:2454] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ccab495427047a296668ea3434ff0d1 in term 1.
I20260925 15:05:38.659412 1746 raft_consensus.cc:2454] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ccab495427047a296668ea3434ff0d1 in term 1.
I20260925 15:05:38.659759 1414 leader_election.cc:302] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccab495427047a296668ea3434ff0d1, 5df334d031b047a2b55da943a2fd37bb; no voters:
I20260925 15:05:38.660017 1817 raft_consensus.cc:2781] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:05:38.660434 1817 raft_consensus.cc:686] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [term 1 LEADER]: Becoming Leader. State: Replica: 3ccab495427047a296668ea3434ff0d1, State: Running, Role: LEADER
I20260925 15:05:38.660853 1817 consensus_queue.cc:237] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f047583bf76440f1b883c8c98d25af4b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38333 } } peers { permanent_uuid: "5df334d031b047a2b55da943a2fd37bb" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44449 } } peers { permanent_uuid: "3ccab495427047a296668ea3434ff0d1" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44455 } }
I20260925 15:05:38.665161 1335 catalog_manager.cc:5495] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ccab495427047a296668ea3434ff0d1 (127.22.236.1). New cstate: current_term: 1 leader_uuid: "3ccab495427047a296668ea3434ff0d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f047583bf76440f1b883c8c98d25af4b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5df334d031b047a2b55da943a2fd37bb" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ccab495427047a296668ea3434ff0d1" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 44455 } health_report { overall_health: HEALTHY } } }
W20260925 15:05:38.675410 1525 tablet.cc:2369] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:05:38.705009 1807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.700002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.22.236.1:44455)
0925 15:05:38.700206 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:38.700212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:38.700281 (+ 69us) server_negotiation.cc:407] Connection header received
0925 15:05:38.700487 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:38.700491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:38.700574 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:38.700708 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:05:38.701051 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.701734 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.703004 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.703398 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.703501 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.703932 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:05:38.704461 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:05:38.704587 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:38.704687 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 15:05:38.704763 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:05:38.712688 1746 raft_consensus.cc:1261] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb [term 1 FOLLOWER]: Refusing update from remote peer 3ccab495427047a296668ea3434ff0d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:05:38.712687 1613 raft_consensus.cc:1261] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b [term 1 FOLLOWER]: Refusing update from remote peer 3ccab495427047a296668ea3434ff0d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:05:38.713814 1820 consensus_queue.cc:1035] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f047583bf76440f1b883c8c98d25af4b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:05:38.714246 1817 consensus_queue.cc:1035] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5df334d031b047a2b55da943a2fd37bb" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:05:38.728197 1825 mvcc.cc:205] Tried to move back new op lower bound from 7333268433757331456 to 7333268433558134784. Current Snapshot: MvccSnapshot[applied={T|T < 7333268433757331456}]
I20260925 15:05:38.734654 1824 mvcc.cc:205] Tried to move back new op lower bound from 7333268433757331456 to 7333268433558134784. Current Snapshot: MvccSnapshot[applied={T|T < 7333268433757331456}]
I20260925 15:05:38.737886 1334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:05:38.738175 1334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:05:38.742012 1334 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5ff295611a04e3d9a1bdd5aaba3b4cd: Sending DeleteTablet for 3 replicas of tablet 49ec0b2fdcc44fce95a2d3c932a7f776
I20260925 15:05:38.743023 1726 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ec0b2fdcc44fce95a2d3c932a7f776 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:38 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:35236
I20260925 15:05:38.743196 1593 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ec0b2fdcc44fce95a2d3c932a7f776 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:38 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:37576
I20260925 15:05:38.743167 1725 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ec0b2fdcc44fce95a2d3c932a7f776 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:38 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:35236
I20260925 15:05:38.743695 1459 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ec0b2fdcc44fce95a2d3c932a7f776 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:05:38 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:56746
I20260925 15:05:38.744813 1832 tablet_replica.cc:330] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb: stopping tablet replica
I20260925 15:05:38.745218 1832 raft_consensus.cc:2229] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:05:38.745528 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 1396
I20260925 15:05:38.745673 1832 raft_consensus.cc:2258] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:38.750463 1832 ts_tablet_manager.cc:1905] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:38.758630 1834 tablet_replica.cc:330] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1: stopping tablet replica
I20260925 15:05:38.759277 1834 raft_consensus.cc:2229] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:05:38.760102 1834 raft_consensus.cc:2258] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:38.763022 1834 ts_tablet_manager.cc:1905] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:38.763477 1832 ts_tablet_manager.cc:1918] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:05:38.763650 1832 log.cc:1196] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/49ec0b2fdcc44fce95a2d3c932a7f776
I20260925 15:05:38.764182 1832 ts_tablet_manager.cc:1939] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 5df334d031b047a2b55da943a2fd37bb: Deleting consensus metadata
Sep 25 15:05:38 dist-test-slave-nsrc krb5kdc[1290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:05:38.768410 1833 tablet_replica.cc:330] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b: stopping tablet replica
I20260925 15:05:38.768752 1833 raft_consensus.cc:2229] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:05:38.769152 1833 raft_consensus.cc:2258] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:05:38.770260 1321 catalog_manager.cc:4841] TS 5df334d031b047a2b55da943a2fd37bb (127.22.236.3:44449): tablet 49ec0b2fdcc44fce95a2d3c932a7f776 (table test-table [id=067ac443c81547ecbf06ccba0d0c9a35]) successfully deleted
I20260925 15:05:38.771070 1833 ts_tablet_manager.cc:1905] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:05:38.772475 1834 ts_tablet_manager.cc:1918] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:05:38.772644 1834 log.cc:1196] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/49ec0b2fdcc44fce95a2d3c932a7f776
W20260925 15:05:38.773342 1321 catalog_manager.cc:4808] TS 5df334d031b047a2b55da943a2fd37bb (127.22.236.3:44449): delete failed for tablet 49ec0b2fdcc44fce95a2d3c932a7f776 because the tablet was not found. No further retry: Not found: Tablet not found: 49ec0b2fdcc44fce95a2d3c932a7f776
I20260925 15:05:38.773456 1834 ts_tablet_manager.cc:1939] T 49ec0b2fdcc44fce95a2d3c932a7f776 P 3ccab495427047a296668ea3434ff0d1: Deleting consensus metadata
I20260925 15:05:38.775122 1321 catalog_manager.cc:4841] TS 3ccab495427047a296668ea3434ff0d1 (127.22.236.1:44455): tablet 49ec0b2fdcc44fce95a2d3c932a7f776 (table test-table [id=067ac443c81547ecbf06ccba0d0c9a35]) successfully deleted
I20260925 15:05:38.777484 1833 ts_tablet_manager.cc:1918] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:05:38.777685 1833 log.cc:1196] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/49ec0b2fdcc44fce95a2d3c932a7f776
I20260925 15:05:38.778179 1833 ts_tablet_manager.cc:1939] T 49ec0b2fdcc44fce95a2d3c932a7f776 P f047583bf76440f1b883c8c98d25af4b: Deleting consensus metadata
I20260925 15:05:38.778668 1807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.748071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.22.236.1:44455)
0925 15:05:38.748262 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:38.748268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:38.748286 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:05:38.748340 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:38.748346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:38.748419 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:38.748524 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:38.757406 (+ 8882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.758718 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.760610 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.761341 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.761436 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.770650 (+ 9214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:38.770668 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:38.770688 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:38.770737 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:38.775376 (+ 4639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:38.776385 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:38.776393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:38.776397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:38.776563 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:38.776965 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:38.776970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:38.776974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:38.777422 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:38.777708 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:38.778081 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 15:05:38.778307 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 15:05:38.779398 1519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:38.779780 1322 catalog_manager.cc:4841] TS f047583bf76440f1b883c8c98d25af4b (127.22.236.2:38333): tablet 49ec0b2fdcc44fce95a2d3c932a7f776 (table test-table [id=067ac443c81547ecbf06ccba0d0c9a35]) successfully deleted
I20260925 15:05:38.875252 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 1396
I20260925 15:05:38.889931 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 1530
Sep 25 15:05:38 dist-test-slave-nsrc krb5kdc[1290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:05:38.904325 1803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:38.890659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37584 (local address 127.22.236.2:38333)
0925 15:05:38.890824 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:38.890830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:38.890848 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:05:38.891036 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:38.891041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:38.891109 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:38.891194 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:38.892375 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.893196 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.894428 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:38.894955 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:38.895061 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:38.899091 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:38.899102 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:38.899115 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:38.899153 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:38.901820 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:38.902403 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:38.902409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:38.902412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:38.902521 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:38.902860 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:38.902864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:38.902866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:38.903252 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:38.903506 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:38.903795 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 15:05:38.903989 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:05:38.905097 1653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:38.991789 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 1530
I20260925 15:05:39.005180 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 1663
Sep 25 15:05:39 dist-test-slave-nsrc krb5kdc[1290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:05:39.018414 1806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:05:39.005854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.22.236.3:44449)
0925 15:05:39.006008 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:05:39.006013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:05:39.006031 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:05:39.006163 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:05:39.006167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:05:39.006230 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:05:39.006308 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:05:39.007485 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:39.008214 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:39.009412 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:05:39.009769 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:05:39.009848 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:05:39.013754 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:05:39.013769 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:05:39.013784 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:05:39.013822 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:05:39.016216 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:39.016764 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:39.016770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:39.016773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:39.016906 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:05:39.017211 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:05:39.017214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:05:39.017216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:05:39.017539 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:05:39.017744 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 15:05:39.017971 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 15:05:39.018166 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:05:39.018987 1786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:39.108603 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 1663
I20260925 15:05:39.120918 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 1303
I20260925 15:05:39.121935 1365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:05:39.225100 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 1303
2026-09-25T15:05:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4001 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:05:39 dist-test-slave-nsrc krb5kdc[1842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:05:39 dist-test-slave-nsrc krb5kdc[1842](info): set up 2 sockets
Sep 25 15:05:39 dist-test-slave-nsrc krb5kdc[1842](info): commencing operation
krb5kdc: starting...
W20260925 15:05:40.224781 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:05:40 dist-test-slave-nsrc krb5kdc[1842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:05:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:05:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.236.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.236.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/rangeradmin_127.22.236.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/rangeradmin_127.22.236.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.236.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.236.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/rangerlookup_127.22.236.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/rangerlookup_127.22.236.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.236.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.236.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/HTTP_127.22.236.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/HTTP_127.22.236.20@KRBTEST.COM.keytab.
I20260925 15:05:40.510713 23472 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSyRird/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/postgres -l logfile start
2026-09-25 15:05:42.793 UTC [1872] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:05:42.793 UTC [1872] LOG: listening on IPv4 address "127.22.236.20", port 50291
2026-09-25 15:05:42.796 UTC [1872] LOG: listening on Unix socket "/tmp/.s.PGSQL.50291"
2026-09-25 15:05:42.816 UTC [1875] LOG: database system was shut down at 2026-09-25 15:05:41 UTC
2026-09-25 15:05:42.819 UTC [1872] LOG: database system is ready to accept connections
I20260925 15:05:43.696554 23472 mini_postgres.cc:96] Postgres bound to 50291
2026-09-25 15:05:43.711 UTC [1883] FATAL: database "slave" does not exist
127.22.236.20:50291 - accepting connections
I20260925 15:05:43.712682 23472 mini_ranger.cc:161] Starting Ranger...
I20260925 15:05:43.736006 23472 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:05:44.165457 23472 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:05:44.165558 23472 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-admin
2026-09-25 15:05:44,453 [I] DB FLAVOR :POSTGRES
2026-09-25 15:05:44,453 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:05:44,453 [I] Checking connection..
2026-09-25 15:05:44,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:05:44,700 [I] Checking connection passed.
2026-09-25 15:05:44,700 [I] --------- Verifying version history table ---------
2026-09-25 15:05:44,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:05:44,946 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:05:44,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:05:45,194 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:05:45,194 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:05:45,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:05:45.421 UTC [1947] WARNING: there is no transaction in progress
2026-09-25 15:05:45,426 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:05:45,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:05:45,674 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:05:45,675 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:05:45,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:05:45,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:05:46,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nsrc.c.gcp-upstream.internal','N') ;"
2026-09-25 15:05:46,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:05:46,630 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:05:46,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:05:46,881 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:05:46,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:05:47,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:05:47,369 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:05:47,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:05:48.260 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.275 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.290 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.305 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.321 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.337 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.387 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.394 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.407 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.417 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.427 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.440 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.449 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.454 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.460 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:48.466 UTC [2055] WARNING: there is no transaction in progress
2026-09-25 15:05:49,048 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:05:49,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:05:49,279 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:05:49,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:05:49,510 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:05:49,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:05:49,764 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:05:49,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:05:50,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:05:50,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nsrc.c.gcp-upstream.internal';"
2026-09-25 15:05:50,487 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:05:50,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:05:50,737 [I] DB_PATCHES have already been applied
I20260925 15:05:50.742616 23472 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSyRird/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:05:50.742718 23472 mini_ranger.cc:193] Using host: 127.22.236.20
I20260925 15:05:50.755934 23472 mini_ranger.cc:227] Ranger admin URL: http://127.22.236.20:48939
Sep 25, 2026 3:05:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:05:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:05:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:05:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:05:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:05:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:05:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:05:51.799305 23472 mini_ranger.cc:160] Time spent starting Ranger: real 8.087s user 0.005s sys 0.030s
Sep 25, 2026 3:06:11 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:06:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 3:06:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:06:11 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:06:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:06:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:06:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:06:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:06:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:06:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:06:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:06:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:06:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:06:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:06:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:06:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:06:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:06:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:06:12.717777 23472 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.236.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.236.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/rangerkms_127.22.236.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/rangerkms_127.22.236.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.236.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.236.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.236.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/HTTP_127.22.236.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.236.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/HTTP_127.22.236.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:06:12 dist-test-slave-nsrc krb5kdc[1842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348772, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:06:12.862264 23472 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:06:12.880659 23472 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:06:13.688791 23472 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:06:13.688957 23472 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-kms
I20260925 15:06:13.689121 23472 mini_ranger_kms.cc:227] Using postgres at 127.22.236.20:50291
2026-09-25 15:06:14,069 [I] DB FLAVOR :POSTGRES
2026-09-25 15:06:14,069 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:06:14,069 [I] Checking connection
2026-09-25 15:06:14,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:06:14,302 [I] connection success
2026-09-25 15:06:14,302 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:06:14,302 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:06:14,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:06:14,521 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:06:14,522 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:06:14,522 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:06:14,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:50291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:06:14,762 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:06:15.219616 23472 mini_ranger_kms.cc:310] Created kms service
I20260925 15:06:15.401213 23472 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:06:15.443709 23472 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:06:15 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348772, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:06:15.811815 23472 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:06:15.812168 23472 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:06:15.812228 23472 mini_ranger_kms.cc:240] Using host: 127.22.236.20
I20260925 15:06:15.831496 23472 mini_ranger_kms.cc:276] Ranger KMS PID: 2235
I20260925 15:06:15.831679 23472 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.236.20:51883
Sep 25, 2026 3:06:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:06:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:06:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:06:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:06:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:06:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 15:06:17.919941 23472 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.058s user 0.103s sys 0.299s
I20260925 15:06:17.920225 23472 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:06:17.920454 23472 mini_ranger_kms.cc:401] 127.22.236.20:51883/kms/v1/keys
Sep 25 15:06:17 dist-test-slave-nsrc krb5kdc[1842](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348777, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.236.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 15:06:18 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348777, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.236.20@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
Sep 25 15:06:18 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348777, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.236.20@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
Sep 25 15:06:21 dist-test-slave-nsrc krb5kdc[1842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:06:21.147708 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:42171
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:42171
--ranger_config_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.236.20:51883/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:06:21.464833 2286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:21.465153 2286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:21.465282 2286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:21.472667 2286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:06:21.472777 2286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:21.472854 2286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:21.472926 2286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:06:21.472985 2286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:06:21.482703 2286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.236.20:51883/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:42171
--ranger_config_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:42171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:06:21.483944 2286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:21.485265 2286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:21.496781 2292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:21.496811 2294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:21.496842 2291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:21.497203 2286 server_base.cc:1034] running on GCE node
I20260925 15:06:21.498137 2286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:21.499496 2286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:21.500682 2286 hybrid_clock.cc:648] HybridClock initialized: now 1790348781500637 us; error 43 us; skew 500 ppm
I20260925 15:06:21.501000 2286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:21 dist-test-slave-nsrc krb5kdc[1842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348781, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:21.508685 2286 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:06:21.510185 2286 webserver.cc:460] Webserver started at http://127.22.236.62:42151/ using document root <none> and password file <none>
I20260925 15:06:21.510807 2286 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:21.510900 2286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:21.511145 2286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:06:21 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348781, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:06:21.559474 2286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "b4c2728a692249e880ab741b74b022bc"
format_stamp: "Formatted at 2026-09-25 15:06:21 on dist-test-slave-nsrc"
server_key: "80c17505062155e8e04cf3ae9acffae9"
server_key_iv: "3c4492f35f21fa6101357a077d7c32f1"
server_key_version: "kuduclusterkey@0"
I20260925 15:06:21.560405 2286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "b4c2728a692249e880ab741b74b022bc"
format_stamp: "Formatted at 2026-09-25 15:06:21 on dist-test-slave-nsrc"
server_key: "80c17505062155e8e04cf3ae9acffae9"
server_key_iv: "3c4492f35f21fa6101357a077d7c32f1"
server_key_version: "kuduclusterkey@0"
I20260925 15:06:21.565702 2286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 15:06:21.584278 2301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:21.585994 2286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:06:21.586220 2286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "b4c2728a692249e880ab741b74b022bc"
format_stamp: "Formatted at 2026-09-25 15:06:21 on dist-test-slave-nsrc"
server_key: "80c17505062155e8e04cf3ae9acffae9"
server_key_iv: "3c4492f35f21fa6101357a077d7c32f1"
server_key_version: "kuduclusterkey@0"
I20260925 15:06:21.586480 2286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:21.609648 2286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:21.614238 2286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:21.614534 2286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:21.633822 2286 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:42171
I20260925 15:06:21.633823 2352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:42171 every 8 connection(s)
I20260925 15:06:21.635457 2286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:06:21.639098 2353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:06:21.640592 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 2286
I20260925 15:06:21.640779 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:06:21.641535 23472 external_mini_cluster.cc:1428] Setting key aaeb5f2f2c0b7fc2ca66d984b0e5d0c3
I20260925 15:06:21.653659 2353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc: Bootstrap starting.
I20260925 15:06:21.657682 2353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc: Neither blocks nor log segments found. Creating new log.
I20260925 15:06:21.658753 2353 log.cc:824] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:06:21.661796 2353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc: No bootstrap required, opened a new log
Sep 25 15:06:21 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:06:21.670189 2353 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c2728a692249e880ab741b74b022bc" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42171 } }
I20260925 15:06:21.670537 2353 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:06:21.670643 2353 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c2728a692249e880ab741b74b022bc, State: Initialized, Role: FOLLOWER
I20260925 15:06:21.671332 2353 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c2728a692249e880ab741b74b022bc" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42171 } }
I20260925 15:06:21.671577 2353 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:06:21.671713 2353 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:06:21.671907 2353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:06:21.674753 2353 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c2728a692249e880ab741b74b022bc" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42171 } }
I20260925 15:06:21.675313 2353 leader_election.cc:302] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c2728a692249e880ab741b74b022bc; no voters:
I20260925 15:06:21.676087 2353 leader_election.cc:288] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:06:21.676133 2358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:06:21.677918 2356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:21.643574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.22.236.62:42171)
0925 15:06:21.656754 (+ 13180us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:21.656770 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:21.657118 (+ 348us) server_negotiation.cc:407] Connection header received
0925 15:06:21.659063 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:21.659085 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:21.659596 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:21.660033 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:21.661157 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:21.663531 (+ 2374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:21.664811 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:21.665840 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:21.665998 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:21.669788 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:21.669796 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:21.669811 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:21.669849 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:21.673328 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:21.674003 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:21.674013 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:21.674022 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:21.674202 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:21.674594 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:21.674598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:21.674601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:21.675314 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:21.675602 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:21.676127 (+ 525us) server_negotiation.cc:299] Negotiation successful
0925 15:06:21.676564 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12910,"thread_start_us":12662,"threads_started":1}
I20260925 15:06:21.678537 2353 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:06:21.684697 2358 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [term 1 LEADER]: Becoming Leader. State: Replica: b4c2728a692249e880ab741b74b022bc, State: Running, Role: LEADER
I20260925 15:06:21.685338 2358 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c2728a692249e880ab741b74b022bc" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42171 } }
I20260925 15:06:21.688098 2359 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4c2728a692249e880ab741b74b022bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c2728a692249e880ab741b74b022bc" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42171 } } }
I20260925 15:06:21.688978 2359 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [sys.catalog]: This master's current role is: LEADER
I20260925 15:06:21.688975 2353 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:06:21.688211 2360 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4c2728a692249e880ab741b74b022bc. Latest consensus state: current_term: 1 leader_uuid: "b4c2728a692249e880ab741b74b022bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c2728a692249e880ab741b74b022bc" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42171 } } }
I20260925 15:06:21.689191 2360 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc [sys.catalog]: This master's current role is: LEADER
I20260925 15:06:21.689958 2373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:06:21.702698 2373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:06:21.712852 2373 catalog_manager.cc:1347] Generated new cluster ID: 4c3b36bf8a3c4a65a21b723abb30b997
I20260925 15:06:21.712970 2373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:06:21.745872 2373 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:06:21.747764 2373 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:06:21.755739 2373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc: Generated new TSK 0
I20260925 15:06:21.756846 2373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:06:22.460696 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.236.20:51883/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:42171
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:06:22 dist-test-slave-nsrc krb5kdc[1842](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348782, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 15:06:22.908946 2398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:22.909441 2398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:22.909651 2398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:22.921291 2398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:22.921450 2398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:22.921738 2398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:06:22.937649 2398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.236.20:51883/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:42171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:06:22.939416 2398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:22.941358 2398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:22.958062 2408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:22.959653 2409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:22.966871 2411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:22.968010 2398 server_base.cc:1034] running on GCE node
I20260925 15:06:22.968834 2398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:22.970069 2398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:22.971356 2398 hybrid_clock.cc:648] HybridClock initialized: now 1790348782971288 us; error 48 us; skew 500 ppm
I20260925 15:06:22.971736 2398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:22 dist-test-slave-nsrc krb5kdc[1842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348782, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:22.975961 2398 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:06:22.978009 2398 webserver.cc:460] Webserver started at http://127.22.236.1:36843/ using document root <none> and password file <none>
I20260925 15:06:22.979092 2398 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:22.979238 2398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:22.979686 2398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:06:22 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348782, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:06:22.999866 2398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "19a3640a808f405a8e44294953af3bd5"
format_stamp: "Formatted at 2026-09-25 15:06:22 on dist-test-slave-nsrc"
server_key: "82809b4ce1e7e5afc9b8e0ac3702f130"
server_key_iv: "00251f9167fa7a9390f8959af6945105"
server_key_version: "kuduclusterkey@0"
I20260925 15:06:23.001561 2398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "19a3640a808f405a8e44294953af3bd5"
format_stamp: "Formatted at 2026-09-25 15:06:22 on dist-test-slave-nsrc"
server_key: "82809b4ce1e7e5afc9b8e0ac3702f130"
server_key_iv: "00251f9167fa7a9390f8959af6945105"
server_key_version: "kuduclusterkey@0"
I20260925 15:06:23.022462 2398 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.005s sys 0.014s
I20260925 15:06:23.055048 2418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:23.056865 2398 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.010s sys 0.000s
I20260925 15:06:23.057065 2398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "19a3640a808f405a8e44294953af3bd5"
format_stamp: "Formatted at 2026-09-25 15:06:22 on dist-test-slave-nsrc"
server_key: "82809b4ce1e7e5afc9b8e0ac3702f130"
server_key_iv: "00251f9167fa7a9390f8959af6945105"
server_key_version: "kuduclusterkey@0"
I20260925 15:06:23.057296 2398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:23.091723 2398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:23.102874 2398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:23.103302 2398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:23.110725 2398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:06:23.125774 2398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:06:23.125969 2398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 15:06:23.126186 2398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:06:23.126317 2398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:06:23 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348782, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:06:23.538501 2398 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:34783
I20260925 15:06:23.538517 2530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:34783 every 8 connection(s)
I20260925 15:06:23.540295 2398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:06:23.548974 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 2398
I20260925 15:06:23.549199 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:06:23.550031 23472 external_mini_cluster.cc:1428] Setting key a8aab166cbcdcf85e392ca861d28db1a
Sep 25 15:06:23 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348782, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:06:23.575690 2533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:23.543276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:58965 (local address 127.22.236.62:42171)
0925 15:06:23.556409 (+ 13133us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:23.556414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:23.556432 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:06:23.556486 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:23.556490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:23.556564 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:23.556659 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:23.558681 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:23.559747 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:23.563161 (+ 3414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:23.563747 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:23.563857 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:23.568741 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:23.568748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:23.568753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:23.568794 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:23.571975 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:23.573003 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:23.573009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:23.573012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:23.573163 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:23.573655 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:23.573659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:23.573662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:23.574042 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:23.574179 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:23.575114 (+ 935us) server_negotiation.cc:299] Negotiation successful
0925 15:06:23.575356 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13016,"thread_start_us":17090,"threads_started":1}
I20260925 15:06:23.578125 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:23.544174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:42171 (local address 127.22.236.1:58965)
0925 15:06:23.552693 (+ 8519us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:23.552747 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:23.554965 (+ 2218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:23.557080 (+ 2115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:23.557092 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:23.557697 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:23.558479 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:23.558504 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:23.560953 (+ 2449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:23.560959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:23.562994 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:23.563009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:23.564120 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:23.564126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:23.564415 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:23.565232 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:23.565263 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:23.568459 (+ 3196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:23.572136 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:23.572170 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:23.572239 (+ 69us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:23.572837 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:23.573290 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:23.573294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:23.573298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:23.573531 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:23.574250 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:23.574258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:23.574694 (+ 436us) client_negotiation.cc:769] Sending connection context
0925 15:06:23.576243 (+ 1549us) client_negotiation.cc:240] Negotiation successful
0925 15:06:23.576772 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8187,"thread_start_us":11066,"threads_started":1}
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
I20260925 15:06:23.580394 2531 heartbeater.cc:347] Connected to a master server at 127.22.236.62:42171
I20260925 15:06:23.580771 2531 heartbeater.cc:464] Registering TS with master...
I20260925 15:06:23.581724 2531 heartbeater.cc:511] Master 127.22.236.62:42171 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
I20260925 15:06:23.584184 2318 ts_manager.cc:194] Registered new tserver with Master: 19a3640a808f405a8e44294953af3bd5 (127.22.236.1:34783)
I20260925 15:06:23.586139 2318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:58965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:06:23.617221 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.236.20:51883/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:42171
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:06:23 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348782, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
W20260925 15:06:24.076712 2537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:24.077090 2537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:24.077240 2537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:24.089038 2537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:24.089178 2537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:24.089357 2537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:06:24.104105 2537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.236.20:51883/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:42171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:06:24.105485 2537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:24.106765 2537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:24.120838 2552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:24.121112 2551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:24.121814 2554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:24.122304 2537 server_base.cc:1034] running on GCE node
I20260925 15:06:24.122853 2537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:24.123819 2537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:24.125015 2537 hybrid_clock.cc:648] HybridClock initialized: now 1790348784124962 us; error 49 us; skew 500 ppm
I20260925 15:06:24.125320 2537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:24 dist-test-slave-nsrc krb5kdc[1842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348784, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:24.128305 2537 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:06:24.129719 2537 webserver.cc:460] Webserver started at http://127.22.236.2:34739/ using document root <none> and password file <none>
I20260925 15:06:24.130373 2537 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:24.130481 2537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:24.130793 2537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:06:24 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348784, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:06:24.151103 2537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "f76775fd17c34448a440a1e1c85ee504"
format_stamp: "Formatted at 2026-09-25 15:06:24 on dist-test-slave-nsrc"
server_key: "4190159cb5baccbd4f066e8acc8964d0"
server_key_iv: "0300895cafa0002e82d5931b42821ee5"
server_key_version: "kuduclusterkey@0"
I20260925 15:06:24.152275 2537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "f76775fd17c34448a440a1e1c85ee504"
format_stamp: "Formatted at 2026-09-25 15:06:24 on dist-test-slave-nsrc"
server_key: "4190159cb5baccbd4f066e8acc8964d0"
server_key_iv: "0300895cafa0002e82d5931b42821ee5"
server_key_version: "kuduclusterkey@0"
I20260925 15:06:24.159950 2537 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260925 15:06:24.177774 2561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:24.179625 2537 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:06:24.179932 2537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "f76775fd17c34448a440a1e1c85ee504"
format_stamp: "Formatted at 2026-09-25 15:06:24 on dist-test-slave-nsrc"
server_key: "4190159cb5baccbd4f066e8acc8964d0"
server_key_iv: "0300895cafa0002e82d5931b42821ee5"
server_key_version: "kuduclusterkey@0"
I20260925 15:06:24.180284 2537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:24.227735 2537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:24.232851 2537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:24.233217 2537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:24.234398 2537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:06:24.236195 2537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:06:24.236325 2537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 15:06:24.236475 2537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:06:24.236572 2537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:24.274850 2537 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:40737
I20260925 15:06:24.274902 2673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:40737 every 8 connection(s)
I20260925 15:06:24.276599 2537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:06:24.285800 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 2537
I20260925 15:06:24.285960 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:06:24.286450 23472 external_mini_cluster.cc:1428] Setting key 6bba3fb69f90e697652c44a0e6a34efa
Sep 25 15:06:24 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348784, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:06:24.316262 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:24.279450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:33675 (local address 127.22.236.62:42171)
0925 15:06:24.299825 (+ 20375us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:24.299830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:24.299846 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:06:24.299904 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:24.299907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:24.299967 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:24.300071 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:24.302048 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.302796 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.304952 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.305366 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.305474 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:24.309447 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:24.309453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:24.309456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:24.309488 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:24.312564 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:24.313444 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:24.313449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:24.313453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:24.313606 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:24.314173 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:24.314178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:24.314182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:24.314551 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:24.314684 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:24.315638 (+ 954us) server_negotiation.cc:299] Negotiation successful
0925 15:06:24.315861 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20268,"thread_start_us":20738,"threads_started":1}
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
I20260925 15:06:24.318553 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:24.280198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:42171 (local address 127.22.236.2:33675)
0925 15:06:24.295160 (+ 14962us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:24.295219 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:24.297795 (+ 2576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:24.300418 (+ 2623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:24.300432 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:24.301039 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:24.301869 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:24.301894 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.302948 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.302953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:24.304821 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:24.304836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.305593 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.305600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:24.305887 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:24.306628 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:24.306656 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:24.309216 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:24.312724 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:24.312733 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:24.312751 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:24.313304 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:24.313753 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:24.313758 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:24.313761 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:24.314038 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:24.314783 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:24.314791 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:24.315204 (+ 413us) client_negotiation.cc:769] Sending connection context
0925 15:06:24.316731 (+ 1527us) client_negotiation.cc:240] Negotiation successful
0925 15:06:24.317261 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14605,"thread_start_us":14358,"threads_started":1}
I20260925 15:06:24.320307 2674 heartbeater.cc:347] Connected to a master server at 127.22.236.62:42171
I20260925 15:06:24.320577 2674 heartbeater.cc:464] Registering TS with master...
I20260925 15:06:24.321190 2674 heartbeater.cc:511] Master 127.22.236.62:42171 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
I20260925 15:06:24.322583 2318 ts_manager.cc:194] Registered new tserver with Master: f76775fd17c34448a440a1e1c85ee504 (127.22.236.2:40737)
I20260925 15:06:24.323588 2318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:33675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:06:24.347501 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.236.20:51883/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:42171
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:06:24.591782 2531 heartbeater.cc:503] Master 127.22.236.62:42171 was elected leader, sending a full tablet report...
W20260925 15:06:24.677425 2680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:24.677704 2680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:24.677821 2680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:24.685421 2680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:24.685600 2680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:24.685755 2680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:06:24.695561 2680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.236.20:51883/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:42171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:06:24.696825 2680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:24.698014 2680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:24.710435 2688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:24.710470 2686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:24.710501 2685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:24.711119 2680 server_base.cc:1034] running on GCE node
I20260925 15:06:24.711712 2680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:24.712635 2680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:24.713814 2680 hybrid_clock.cc:648] HybridClock initialized: now 1790348784713772 us; error 32 us; skew 500 ppm
I20260925 15:06:24.714092 2680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:24 dist-test-slave-nsrc krb5kdc[1842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348784, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:24.717032 2680 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:06:24.718331 2680 webserver.cc:460] Webserver started at http://127.22.236.3:42001/ using document root <none> and password file <none>
I20260925 15:06:24.718942 2680 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:24.719064 2680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:24.719340 2680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:06:24 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348784, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:06:24.732321 2680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "1606f20f3e7c45fcb4fb954142fdab6e"
format_stamp: "Formatted at 2026-09-25 15:06:24 on dist-test-slave-nsrc"
server_key: "9ee1896527da6f1c28b527db0eafe0e1"
server_key_iv: "3d9e1a7579f5439a1375d935658c9742"
server_key_version: "kuduclusterkey@0"
I20260925 15:06:24.733104 2680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "1606f20f3e7c45fcb4fb954142fdab6e"
format_stamp: "Formatted at 2026-09-25 15:06:24 on dist-test-slave-nsrc"
server_key: "9ee1896527da6f1c28b527db0eafe0e1"
server_key_iv: "3d9e1a7579f5439a1375d935658c9742"
server_key_version: "kuduclusterkey@0"
I20260925 15:06:24.738234 2680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:06:24.754642 2695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:24.756129 2680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:06:24.756335 2680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "1606f20f3e7c45fcb4fb954142fdab6e"
format_stamp: "Formatted at 2026-09-25 15:06:24 on dist-test-slave-nsrc"
server_key: "9ee1896527da6f1c28b527db0eafe0e1"
server_key_iv: "3d9e1a7579f5439a1375d935658c9742"
server_key_version: "kuduclusterkey@0"
I20260925 15:06:24.756567 2680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:24.771646 2680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:24.775358 2680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:24.775602 2680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:24.776432 2680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:06:24.777719 2680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:06:24.777786 2680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:24.777870 2680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:06:24.777915 2680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:24.807792 2680 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:35763
I20260925 15:06:24.807834 2807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:35763 every 8 connection(s)
I20260925 15:06:24.809023 2680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:06:24.817744 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 2680
I20260925 15:06:24.817884 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:06:24.818324 23472 external_mini_cluster.cc:1428] Setting key b4cba34f0df04536029f0df12485cacb
Sep 25 15:06:24 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348784, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:06:24.826618 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:24.811724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:50081 (local address 127.22.236.62:42171)
0925 15:06:24.811915 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:24.811921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:24.812951 (+ 1030us) server_negotiation.cc:407] Connection header received
0925 15:06:24.814534 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:24.814539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:24.814614 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:24.814698 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:24.815925 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.816679 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.818166 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.818587 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.818696 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:24.822062 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:24.822067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:24.822070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:24.822100 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:24.824318 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:24.824895 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:24.824899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:24.824902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:24.825025 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:24.825353 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:24.825356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:24.825358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:24.825593 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:24.825701 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:24.826248 (+ 547us) server_negotiation.cc:299] Negotiation successful
0925 15:06:24.826404 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:06:24.827883 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:24.812179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:42171 (local address 127.22.236.3:50081)
0925 15:06:24.812801 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:24.812840 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:24.814304 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:24.814875 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:24.814882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:24.815265 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:24.815789 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:24.815806 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.816790 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.816794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:24.818042 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:24.818053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.818790 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.818796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:24.819002 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:24.819672 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:24.819700 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:24.821888 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:24.824420 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:24.824426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:24.824437 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:24.824808 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:24.825101 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:24.825104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:24.825106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:24.825278 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:24.825700 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:24.825706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:24.825966 (+ 260us) client_negotiation.cc:769] Sending connection context
0925 15:06:24.826191 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 15:06:24.826519 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":238,"threads_started":1}
I20260925 15:06:24.829064 2808 heartbeater.cc:347] Connected to a master server at 127.22.236.62:42171
I20260925 15:06:24.829319 2808 heartbeater.cc:464] Registering TS with master...
I20260925 15:06:24.829937 2808 heartbeater.cc:511] Master 127.22.236.62:42171 requested a full tablet report, sending...
I20260925 15:06:24.831157 2318 ts_manager.cc:194] Registered new tserver with Master: 1606f20f3e7c45fcb4fb954142fdab6e (127.22.236.3:35763)
I20260925 15:06:24.831924 2318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:50081
I20260925 15:06:24.833319 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:06:24.849159 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:24.837436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.22.236.62:42171)
0925 15:06:24.837568 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:24.837573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:24.838006 (+ 433us) server_negotiation.cc:407] Connection header received
0925 15:06:24.838158 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:24.838162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:24.838227 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:24.838301 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:24.839534 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.840258 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.841666 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.842078 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.842165 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:24.844517 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:24.844524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:24.844528 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:24.844564 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:24.846989 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:24.847606 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:24.847610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:24.847612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:24.847725 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:24.848050 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:24.848053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:24.848055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:24.848338 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:24.848535 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:24.848770 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 15:06:24.848917 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:06:24.853353 2318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:06:24.860060 2318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:06:24.897132 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:24.873092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.22.236.2:40737)
0925 15:06:24.888960 (+ 15868us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:24.888967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:24.888990 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:06:24.889060 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:24.889064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:24.889265 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:24.889581 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:24.890380 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.891816 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.894977 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.895958 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.896063 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:24.896250 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:24.896630 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 15:06:24.896790 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15742,"thread_start_us":15453,"threads_started":1}
I20260925 15:06:24.897198 2822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:24.872706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:40737 (local address 127.0.0.1:50418)
0925 15:06:24.887858 (+ 15152us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:24.887883 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:24.888017 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:24.889470 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:24.889489 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:24.889509 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:24.890244 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:24.890258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.891957 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.891961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:24.894857 (+ 2896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:24.894868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.896172 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.896179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:24.896389 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:24.896441 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 15:06:24.896579 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 15:06:24.896787 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15022,"mutex_wait_us":148,"thread_start_us":15540,"threads_started":1}
I20260925 15:06:24.897406 2823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:24.873390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.22.236.1:34783)
0925 15:06:24.887396 (+ 14006us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:24.887407 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:24.888568 (+ 1161us) server_negotiation.cc:407] Connection header received
0925 15:06:24.888700 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:24.888709 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:24.889007 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:24.889244 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:24.890038 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.891767 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.895168 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.896083 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.896264 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:24.896476 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:24.896854 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 15:06:24.897085 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13810,"thread_start_us":13642,"threads_started":1}
I20260925 15:06:24.898069 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:24.872786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:34783 (local address 127.0.0.1:55422)
0925 15:06:24.888453 (+ 15667us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:24.888468 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:24.888564 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:24.889259 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:24.889267 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:24.889301 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:24.889803 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:24.889817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.891896 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.891902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:24.895013 (+ 3111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:24.895027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.896330 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.896337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:24.896519 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:24.896555 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:06:24.896650 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:06:24.897674 (+ 1024us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15572,"mutex_wait_us":165,"thread_start_us":14778,"threads_started":1}
I20260925 15:06:24.900743 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:24.871914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:35763 (local address 127.0.0.1:57116)
0925 15:06:24.880667 (+ 8753us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:24.880705 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:24.880883 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:24.892303 (+ 11420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:24.892308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:24.892335 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:24.893236 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:24.893248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.896447 (+ 3199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.896452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:24.898477 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:24.898502 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.900006 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.900013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:24.900211 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:24.900244 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:06:24.900322 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:06:24.900448 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8492,"thread_start_us":8360,"threads_started":1}
I20260925 15:06:24.900868 2824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:24.873242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.22.236.3:35763)
0925 15:06:24.890180 (+ 16938us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:24.890189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:24.890219 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:06:24.890306 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:24.890312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:24.892118 (+ 1806us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:24.892310 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:24.893402 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.896302 (+ 2900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.898675 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:24.899804 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:24.899916 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:24.900135 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:24.900390 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 15:06:24.900566 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16777,"thread_start_us":16640,"threads_started":1}
I20260925 15:06:24.901559 2609 tablet_service.cc:1467] Processing CreateTablet for tablet fff2113577c64e49a64346134d301933 (DEFAULT_TABLE table=test-table [id=65a0af87519d435882bc51d617d8548c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:06:24.902357 2466 tablet_service.cc:1467] Processing CreateTablet for tablet fff2113577c64e49a64346134d301933 (DEFAULT_TABLE table=test-table [id=65a0af87519d435882bc51d617d8548c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:06:24.903177 2609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff2113577c64e49a64346134d301933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:06:24.903741 2466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff2113577c64e49a64346134d301933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:06:24.903789 2743 tablet_service.cc:1467] Processing CreateTablet for tablet fff2113577c64e49a64346134d301933 (DEFAULT_TABLE table=test-table [id=65a0af87519d435882bc51d617d8548c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:06:24.905038 2743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff2113577c64e49a64346134d301933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:06:24.916415 2826 tablet_bootstrap.cc:492] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e: Bootstrap starting.
I20260925 15:06:24.920516 2826 tablet_bootstrap.cc:654] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e: Neither blocks nor log segments found. Creating new log.
I20260925 15:06:24.921439 2826 log.cc:824] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:06:24.923002 2827 tablet_bootstrap.cc:492] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504: Bootstrap starting.
I20260925 15:06:24.924592 2828 tablet_bootstrap.cc:492] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5: Bootstrap starting.
I20260925 15:06:24.926431 2827 tablet_bootstrap.cc:654] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504: Neither blocks nor log segments found. Creating new log.
I20260925 15:06:24.927583 2827 log.cc:824] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504: Log is configured to *not* fsync() on all Append() calls
I20260925 15:06:24.928802 2828 tablet_bootstrap.cc:654] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5: Neither blocks nor log segments found. Creating new log.
I20260925 15:06:24.930349 2828 log.cc:824] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:06:24.930884 2827 tablet_bootstrap.cc:492] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504: No bootstrap required, opened a new log
I20260925 15:06:24.931136 2827 ts_tablet_manager.cc:1397] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260925 15:06:24.932866 2826 tablet_bootstrap.cc:492] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e: No bootstrap required, opened a new log
I20260925 15:06:24.933107 2826 ts_tablet_manager.cc:1397] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20260925 15:06:24.933879 2828 tablet_bootstrap.cc:492] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5: No bootstrap required, opened a new log
I20260925 15:06:24.934116 2828 ts_tablet_manager.cc:1397] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260925 15:06:24.940251 2826 raft_consensus.cc:348] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35763 } } peers { permanent_uuid: "19a3640a808f405a8e44294953af3bd5" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34783 } } peers { permanent_uuid: "f76775fd17c34448a440a1e1c85ee504" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40737 } }
I20260925 15:06:24.940590 2826 raft_consensus.cc:374] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:06:24.940711 2826 raft_consensus.cc:729] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1606f20f3e7c45fcb4fb954142fdab6e, State: Initialized, Role: FOLLOWER
I20260925 15:06:24.941067 2827 raft_consensus.cc:348] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35763 } } peers { permanent_uuid: "19a3640a808f405a8e44294953af3bd5" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34783 } } peers { permanent_uuid: "f76775fd17c34448a440a1e1c85ee504" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40737 } }
I20260925 15:06:24.941612 2827 raft_consensus.cc:374] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:06:24.941718 2827 raft_consensus.cc:729] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76775fd17c34448a440a1e1c85ee504, State: Initialized, Role: FOLLOWER
I20260925 15:06:24.941371 2826 consensus_queue.cc:260] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35763 } } peers { permanent_uuid: "19a3640a808f405a8e44294953af3bd5" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34783 } } peers { permanent_uuid: "f76775fd17c34448a440a1e1c85ee504" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40737 } }
I20260925 15:06:24.942390 2827 consensus_queue.cc:260] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35763 } } peers { permanent_uuid: "19a3640a808f405a8e44294953af3bd5" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34783 } } peers { permanent_uuid: "f76775fd17c34448a440a1e1c85ee504" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40737 } }
I20260925 15:06:24.942839 2808 heartbeater.cc:503] Master 127.22.236.62:42171 was elected leader, sending a full tablet report...
I20260925 15:06:24.943800 2828 raft_consensus.cc:348] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35763 } } peers { permanent_uuid: "19a3640a808f405a8e44294953af3bd5" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34783 } } peers { permanent_uuid: "f76775fd17c34448a440a1e1c85ee504" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40737 } }
I20260925 15:06:24.944063 2674 heartbeater.cc:503] Master 127.22.236.62:42171 was elected leader, sending a full tablet report...
I20260925 15:06:24.944098 2827 ts_tablet_manager.cc:1428] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260925 15:06:24.944260 2828 raft_consensus.cc:374] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:06:24.944375 2828 raft_consensus.cc:729] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a3640a808f405a8e44294953af3bd5, State: Initialized, Role: FOLLOWER
I20260925 15:06:24.945091 2826 ts_tablet_manager.cc:1428] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260925 15:06:24.945008 2828 consensus_queue.cc:260] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35763 } } peers { permanent_uuid: "19a3640a808f405a8e44294953af3bd5" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34783 } } peers { permanent_uuid: "f76775fd17c34448a440a1e1c85ee504" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40737 } }
I20260925 15:06:24.947062 2828 ts_tablet_manager.cc:1428] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
W20260925 15:06:25.034919 2675 tablet.cc:2369] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:06:25.060843 2809 tablet.cc:2369] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:06:25.062031 2532 tablet.cc:2369] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:06:25.205821 2832 raft_consensus.cc:482] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:06:25.206077 2832 raft_consensus.cc:504] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35763 } } peers { permanent_uuid: "19a3640a808f405a8e44294953af3bd5" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34783 } } peers { permanent_uuid: "f76775fd17c34448a440a1e1c85ee504" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40737 } }
I20260925 15:06:25.207640 2832 leader_election.cc:288] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19a3640a808f405a8e44294953af3bd5 (127.22.236.1:34783), f76775fd17c34448a440a1e1c85ee504 (127.22.236.2:40737)
I20260925 15:06:25.213749 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:25.208239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:38729 (local address 127.22.236.2:40737)
0925 15:06:25.208413 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:25.208418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:25.208435 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:06:25.208503 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:25.208508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:25.208579 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:25.208654 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:25.209060 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.209952 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.211851 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.213020 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.213130 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:25.213183 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:25.213473 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 15:06:25.213557 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:06:25.213850 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:25.207910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:40737 (local address 127.22.236.3:38729)
0925 15:06:25.208240 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:25.208259 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:25.208374 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:25.208701 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:25.208709 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:25.208734 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:25.208897 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:25.208907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.210090 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.210095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:25.211705 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:25.211717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.213155 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.213162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:25.213335 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:25.213369 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:06:25.213463 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:06:25.213591 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260925 15:06:25.214915 2823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:25.208121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:60271 (local address 127.22.236.1:34783)
0925 15:06:25.208373 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:25.208379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:25.208936 (+ 557us) server_negotiation.cc:407] Connection header received
0925 15:06:25.209054 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:25.209059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:25.209117 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:25.209178 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:25.209691 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.210611 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.213061 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.214190 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.214336 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:25.214401 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:25.214669 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 15:06:25.214754 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 15:06:25.215046 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:25.207973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:34783 (local address 127.22.236.3:60271)
0925 15:06:25.208827 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:25.208846 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:25.208946 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:25.209216 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:25.209220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:25.209244 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:25.209563 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:25.209573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.210747 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.210751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:25.212917 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:25.212931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.214377 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.214384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:25.214523 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:25.214554 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:06:25.214644 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:06:25.214756 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":535,"threads_started":1}
I20260925 15:06:25.214960 2629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff2113577c64e49a64346134d301933" candidate_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76775fd17c34448a440a1e1c85ee504" is_pre_election: true
I20260925 15:06:25.215525 2629 raft_consensus.cc:2454] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1606f20f3e7c45fcb4fb954142fdab6e in term 0.
I20260925 15:06:25.215942 2486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff2113577c64e49a64346134d301933" candidate_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a3640a808f405a8e44294953af3bd5" is_pre_election: true
I20260925 15:06:25.216106 2699 leader_election.cc:302] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1606f20f3e7c45fcb4fb954142fdab6e, f76775fd17c34448a440a1e1c85ee504; no voters:
I20260925 15:06:25.216393 2486 raft_consensus.cc:2454] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1606f20f3e7c45fcb4fb954142fdab6e in term 0.
I20260925 15:06:25.216475 2832 raft_consensus.cc:2781] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:06:25.216602 2832 raft_consensus.cc:482] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:06:25.216686 2832 raft_consensus.cc:3037] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:06:25.218679 2832 raft_consensus.cc:504] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35763 } } peers { permanent_uuid: "19a3640a808f405a8e44294953af3bd5" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34783 } } peers { permanent_uuid: "f76775fd17c34448a440a1e1c85ee504" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40737 } }
I20260925 15:06:25.219213 2832 leader_election.cc:288] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [CANDIDATE]: Term 1 election: Requested vote from peers 19a3640a808f405a8e44294953af3bd5 (127.22.236.1:34783), f76775fd17c34448a440a1e1c85ee504 (127.22.236.2:40737)
I20260925 15:06:25.219549 2486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff2113577c64e49a64346134d301933" candidate_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a3640a808f405a8e44294953af3bd5"
I20260925 15:06:25.219614 2629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff2113577c64e49a64346134d301933" candidate_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76775fd17c34448a440a1e1c85ee504"
I20260925 15:06:25.219715 2486 raft_consensus.cc:3037] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:06:25.219775 2629 raft_consensus.cc:3037] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:06:25.221773 2629 raft_consensus.cc:2454] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1606f20f3e7c45fcb4fb954142fdab6e in term 1.
I20260925 15:06:25.221787 2486 raft_consensus.cc:2454] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1606f20f3e7c45fcb4fb954142fdab6e in term 1.
I20260925 15:06:25.222088 2699 leader_election.cc:302] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1606f20f3e7c45fcb4fb954142fdab6e, f76775fd17c34448a440a1e1c85ee504; no voters:
I20260925 15:06:25.222326 2832 raft_consensus.cc:2781] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:06:25.223115 2832 raft_consensus.cc:686] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [term 1 LEADER]: Becoming Leader. State: Replica: 1606f20f3e7c45fcb4fb954142fdab6e, State: Running, Role: LEADER
I20260925 15:06:25.223570 2832 consensus_queue.cc:237] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35763 } } peers { permanent_uuid: "19a3640a808f405a8e44294953af3bd5" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34783 } } peers { permanent_uuid: "f76775fd17c34448a440a1e1c85ee504" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40737 } }
I20260925 15:06:25.227649 2316 catalog_manager.cc:5495] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1606f20f3e7c45fcb4fb954142fdab6e (127.22.236.3). New cstate: current_term: 1 leader_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1606f20f3e7c45fcb4fb954142fdab6e" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a3640a808f405a8e44294953af3bd5" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f76775fd17c34448a440a1e1c85ee504" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40737 } health_report { overall_health: UNKNOWN } } }
I20260925 15:06:25.266161 2824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:25.261400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.22.236.3:35763)
0925 15:06:25.261537 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:25.261541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:25.261559 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:06:25.261760 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:25.261764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:25.261843 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:25.261958 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:06:25.262278 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.262949 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.264257 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.264642 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.264731 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:25.265134 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:06:25.265666 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:06:25.265784 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:25.265906 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 15:06:25.265968 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:06:25.273566 2629 raft_consensus.cc:1261] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504 [term 1 FOLLOWER]: Refusing update from remote peer 1606f20f3e7c45fcb4fb954142fdab6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:06:25.273583 2486 raft_consensus.cc:1261] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5 [term 1 FOLLOWER]: Refusing update from remote peer 1606f20f3e7c45fcb4fb954142fdab6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:06:25.274464 2836 consensus_queue.cc:1035] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a3640a808f405a8e44294953af3bd5" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:06:25.274839 2832 consensus_queue.cc:1035] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f76775fd17c34448a440a1e1c85ee504" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:06:25.292443 2840 mvcc.cc:205] Tried to move back new op lower bound from 7333268624471719936 to 7333268624279257088. Current Snapshot: MvccSnapshot[applied={T|T < 7333268624471719936}]
I20260925 15:06:25.311066 2316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:06:25.311329 2316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:06:25.314754 2316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4c2728a692249e880ab741b74b022bc: Sending DeleteTablet for 3 replicas of tablet fff2113577c64e49a64346134d301933
I20260925 15:06:25.315819 2743 tablet_service.cc:1514] Processing DeleteTablet for tablet fff2113577c64e49a64346134d301933 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:06:25 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:57116
I20260925 15:06:25.315893 2466 tablet_service.cc:1514] Processing DeleteTablet for tablet fff2113577c64e49a64346134d301933 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:06:25 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:55422
I20260925 15:06:25.316172 2609 tablet_service.cc:1514] Processing DeleteTablet for tablet fff2113577c64e49a64346134d301933 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:06:25 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:50418
I20260925 15:06:25.318395 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 2398
I20260925 15:06:25.345960 2851 tablet_replica.cc:330] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504: stopping tablet replica
I20260925 15:06:25.346527 2851 raft_consensus.cc:2229] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:06:25.347775 2850 tablet_replica.cc:330] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5: stopping tablet replica
I20260925 15:06:25.347977 2851 raft_consensus.cc:2258] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:06:25.348253 2850 raft_consensus.cc:2229] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:06:25.348783 2850 raft_consensus.cc:2258] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:06:25.349225 2849 tablet_replica.cc:330] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e: stopping tablet replica
I20260925 15:06:25.349619 2849 raft_consensus.cc:2229] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:06:25.350276 2849 raft_consensus.cc:2258] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:06:25.350731 2851 ts_tablet_manager.cc:1905] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:06:25.352191 2849 ts_tablet_manager.cc:1905] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 15:06:25 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:06:25.357306 2851 ts_tablet_manager.cc:1918] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:06:25.357311 2850 ts_tablet_manager.cc:1905] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:06:25.357688 2851 log.cc:1196] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/fff2113577c64e49a64346134d301933
I20260925 15:06:25.358173 2851 ts_tablet_manager.cc:1939] T fff2113577c64e49a64346134d301933 P f76775fd17c34448a440a1e1c85ee504: Deleting consensus metadata
I20260925 15:06:25.359493 2849 ts_tablet_manager.cc:1918] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:06:25.359676 2849 log.cc:1196] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/fff2113577c64e49a64346134d301933
I20260925 15:06:25.360234 2849 ts_tablet_manager.cc:1939] T fff2113577c64e49a64346134d301933 P 1606f20f3e7c45fcb4fb954142fdab6e: Deleting consensus metadata
I20260925 15:06:25.360419 2305 catalog_manager.cc:4841] TS f76775fd17c34448a440a1e1c85ee504 (127.22.236.2:40737): tablet fff2113577c64e49a64346134d301933 (table test-table [id=65a0af87519d435882bc51d617d8548c]) successfully deleted
I20260925 15:06:25.361851 2302 catalog_manager.cc:4841] TS 1606f20f3e7c45fcb4fb954142fdab6e (127.22.236.3:35763): tablet fff2113577c64e49a64346134d301933 (table test-table [id=65a0af87519d435882bc51d617d8548c]) successfully deleted
I20260925 15:06:25.365672 2850 ts_tablet_manager.cc:1918] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:06:25.365823 2850 log.cc:1196] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/fff2113577c64e49a64346134d301933
I20260925 15:06:25.366219 2850 ts_tablet_manager.cc:1939] T fff2113577c64e49a64346134d301933 P 19a3640a808f405a8e44294953af3bd5: Deleting consensus metadata
I20260925 15:06:25.367439 2823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:25.319784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55430 (local address 127.22.236.1:34783)
0925 15:06:25.319953 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:25.319959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:25.342934 (+ 22975us) server_negotiation.cc:407] Connection header received
0925 15:06:25.343440 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:25.343446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:25.343534 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:25.343611 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:25.347144 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.348367 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.350559 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.351161 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.351258 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:25.359280 (+ 8022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:25.359295 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:25.359313 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:25.359365 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:25.364158 (+ 4793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:25.365010 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:25.365019 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:25.365023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:25.365201 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:25.365654 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:25.365663 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:25.365667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:25.366140 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:25.366449 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:25.366776 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 15:06:25.367034 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:06:25.367933 2304 catalog_manager.cc:4841] TS 19a3640a808f405a8e44294953af3bd5 (127.22.236.1:34783): tablet fff2113577c64e49a64346134d301933 (table test-table [id=65a0af87519d435882bc51d617d8548c]) successfully deleted
I20260925 15:06:25.368112 2526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:06:25.486658 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 2398
I20260925 15:06:25.507375 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 2537
Sep 25 15:06:25 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:06:25.523645 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:25.508122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.22.236.2:40737)
0925 15:06:25.508301 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:25.508305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:25.508377 (+ 72us) server_negotiation.cc:407] Connection header received
0925 15:06:25.508556 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:25.508562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:25.508637 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:25.508758 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:25.510330 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.511012 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.512841 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.513211 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.513308 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:25.518333 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:25.518347 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:25.518363 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:25.518395 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:25.521114 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:25.521857 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:25.521863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:25.521866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:25.521981 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:25.522341 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:25.522343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:25.522345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:25.522691 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:25.522892 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:25.523222 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 15:06:25.523375 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:06:25.524338 2669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:06:25.611317 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 2537
I20260925 15:06:25.624217 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 2680
Sep 25 15:06:25 dist-test-slave-nsrc krb5kdc[1842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:06:25.643260 2824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:25.624988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.22.236.3:35763)
0925 15:06:25.625141 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:25.625145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:25.625217 (+ 72us) server_negotiation.cc:407] Connection header received
0925 15:06:25.625393 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:25.625397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:25.625461 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:25.625607 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:25.627407 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.628261 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.630069 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:25.630506 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:25.630614 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:25.636048 (+ 5434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:25.636072 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:25.636092 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:25.636253 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:25.640116 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:25.640952 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:25.640962 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:25.640965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:25.641170 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:25.641580 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:25.641585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:25.641588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:25.642007 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:25.642325 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:25.642608 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 15:06:25.642826 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:06:25.643941 2803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:06:25.763708 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 2680
I20260925 15:06:25.778358 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 2286
I20260925 15:06:25.779536 2348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:06:25.886669 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 2286
I20260925 15:06:25.906184 23472 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:06:26.240661 23472 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:06:26.240868 23472 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:06:26.574604 23472 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:06:26.574 UTC [1872] LOG: received smart shutdown request
2026-09-25 15:06:26.578 UTC [1872] LOG: background worker "logical replication launcher" (PID 1881) exited with exit code 1
2026-09-25 15:06:26.578 UTC [1876] LOG: shutting down
2026-09-25 15:06:26.727 UTC [1872] LOG: database system is shut down
2026-09-25T15:06:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47608 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:06:26 dist-test-slave-nsrc krb5kdc[2865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:06:26 dist-test-slave-nsrc krb5kdc[2865](info): set up 2 sockets
Sep 25 15:06:26 dist-test-slave-nsrc krb5kdc[2865](info): commencing operation
krb5kdc: starting...
W20260925 15:06:27.848378 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:06:27 dist-test-slave-nsrc krb5kdc[2865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:06:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:06:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.236.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.236.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/rangeradmin_127.22.236.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/rangeradmin_127.22.236.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.236.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.236.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/rangerlookup_127.22.236.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/rangerlookup_127.22.236.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.236.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.236.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/HTTP_127.22.236.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/HTTP_127.22.236.20@KRBTEST.COM.keytab.
I20260925 15:06:28.126775 23472 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSyRird/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/postgres -l logfile start
2026-09-25 15:06:30.429 UTC [2895] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:06:30.429 UTC [2895] LOG: listening on IPv4 address "127.22.236.20", port 48929
2026-09-25 15:06:30.432 UTC [2895] LOG: listening on Unix socket "/tmp/.s.PGSQL.48929"
2026-09-25 15:06:30.452 UTC [2898] LOG: database system was shut down at 2026-09-25 15:06:29 UTC
2026-09-25 15:06:30.455 UTC [2895] LOG: database system is ready to accept connections
I20260925 15:06:31.344036 23472 mini_postgres.cc:96] Postgres bound to 48929
2026-09-25 15:06:31.359 UTC [2906] FATAL: database "slave" does not exist
127.22.236.20:48929 - accepting connections
I20260925 15:06:31.359617 23472 mini_ranger.cc:161] Starting Ranger...
I20260925 15:06:31.382352 23472 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:06:31.821297 23472 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:06:31.821417 23472 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-admin
2026-09-25 15:06:32,090 [I] DB FLAVOR :POSTGRES
2026-09-25 15:06:32,091 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:06:32,091 [I] Checking connection..
2026-09-25 15:06:32,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:06:32,324 [I] Checking connection passed.
2026-09-25 15:06:32,325 [I] --------- Verifying version history table ---------
2026-09-25 15:06:32,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:06:32,572 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:06:32,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:06:32,822 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:06:32,822 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:06:32,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:06:33.058 UTC [2970] WARNING: there is no transaction in progress
2026-09-25 15:06:33,063 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:06:33,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:06:33,316 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:06:33,316 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:06:33,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:06:33,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:06:33,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nsrc.c.gcp-upstream.internal','N') ;"
2026-09-25 15:06:33,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:06:34,242 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:06:34,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:06:34,494 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:06:34,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:06:34,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:06:34,986 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:06:34,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:06:35.884 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:35.899 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:35.914 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:35.929 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:35.943 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:35.959 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:36.020 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:36.027 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:36.041 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:36.051 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:36.061 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:36.074 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:36.083 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:36.090 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:36.097 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:36.104 UTC [3078] WARNING: there is no transaction in progress
2026-09-25 15:06:36,717 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:06:36,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:06:36,951 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:06:36,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:06:37,203 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:06:37,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:06:37,451 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:06:37,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:06:37,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:06:37,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nsrc.c.gcp-upstream.internal';"
2026-09-25 15:06:38,182 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:06:38,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:06:38,425 [I] DB_PATCHES have already been applied
I20260925 15:06:38.430598 23472 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSyRird/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:06:38.430689 23472 mini_ranger.cc:193] Using host: 127.22.236.20
I20260925 15:06:38.443342 23472 mini_ranger.cc:227] Ranger admin URL: http://127.22.236.20:44733
Sep 25, 2026 3:06:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:06:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:06:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:06:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:06:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:06:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:06:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:06:39.445767 23472 mini_ranger.cc:160] Time spent starting Ranger: real 8.086s user 0.000s sys 0.035s
Sep 25, 2026 3:06:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:06:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 3:06:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:06:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:06:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:07:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:07:00.797598 23472 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.236.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.236.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/rangerkms_127.22.236.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.236.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/rangerkms_127.22.236.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.236.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.236.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.236.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/HTTP_127.22.236.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.236.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/HTTP_127.22.236.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:07:00 dist-test-slave-nsrc krb5kdc[2865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348820, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:07:00.949961 23472 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:07:00.966850 23472 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:07:01.722921 23472 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:07:01.723032 23472 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-kms
I20260925 15:07:01.723071 23472 mini_ranger_kms.cc:227] Using postgres at 127.22.236.20:48929
2026-09-25 15:07:02,045 [I] DB FLAVOR :POSTGRES
2026-09-25 15:07:02,045 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:07:02,045 [I] Checking connection
2026-09-25 15:07:02,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:07:02,279 [I] connection success
2026-09-25 15:07:02,279 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:07:02,280 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:07:02,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:07:02,506 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:07:02,506 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:07:02,506 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:07:02,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.236.20:48929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:07:02,754 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:07:03.212517 23472 mini_ranger_kms.cc:310] Created kms service
I20260925 15:07:03.439651 23472 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:07:03.482705 23472 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:07:03 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348820, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:07:03.893639 23472 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:07:03.893989 23472 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSyRird/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSyRird/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:07:03.894059 23472 mini_ranger_kms.cc:240] Using host: 127.22.236.20
I20260925 15:07:03.907294 23472 mini_ranger_kms.cc:276] Ranger KMS PID: 3258
I20260925 15:07:03.907500 23472 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.236.20:34561
Sep 25, 2026 3:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:07:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:07:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 15:07:05.979938 23472 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.030s user 0.073s sys 0.262s
I20260925 15:07:05.980224 23472 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:07:05.980422 23472 mini_ranger_kms.cc:401] 127.22.236.20:34561/kms/v1/keys
Sep 25 15:07:06 dist-test-slave-nsrc krb5kdc[2865](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348826, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.236.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 15:07:06 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348826, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.236.20@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
Sep 25 15:07:06 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348826, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.236.20@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
Sep 25 15:07:09 dist-test-slave-nsrc krb5kdc[2865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:07:09.347729 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:33217
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:33217
--ranger_config_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.236.20:34561/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:09.683408 3309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:09.683710 3309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:09.683866 3309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:09.691521 3309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:09.691650 3309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:09.691705 3309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:09.691783 3309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:09.691844 3309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:09.702345 3309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.236.20:34561/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:33217
--ranger_config_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:33217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:09.703679 3309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:09.705075 3309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:09.718508 3314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:09.718586 3315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:09.718798 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:09.718863 3309 server_base.cc:1034] running on GCE node
I20260925 15:07:09.719920 3309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:09.721393 3309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:09.722613 3309 hybrid_clock.cc:648] HybridClock initialized: now 1790348829722573 us; error 32 us; skew 500 ppm
I20260925 15:07:09.722987 3309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:09 dist-test-slave-nsrc krb5kdc[2865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348829, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:09.731794 3309 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:07:09.733453 3309 webserver.cc:460] Webserver started at http://127.22.236.62:46293/ using document root <none> and password file <none>
I20260925 15:07:09.734110 3309 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:09.734208 3309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:09.734500 3309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:07:09 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348829, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:07:09.799085 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "2cdf78aff1bf4d5795d1128d3942c8cf"
format_stamp: "Formatted at 2026-09-25 15:07:09 on dist-test-slave-nsrc"
server_key: "dc9c443f52d7a02c672729c1dd83ea89"
server_key_iv: "4df3f97d771019f7b77b8ef13882b3fe"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:09.800218 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "2cdf78aff1bf4d5795d1128d3942c8cf"
format_stamp: "Formatted at 2026-09-25 15:07:09 on dist-test-slave-nsrc"
server_key: "dc9c443f52d7a02c672729c1dd83ea89"
server_key_iv: "4df3f97d771019f7b77b8ef13882b3fe"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:09.807382 3309 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260925 15:07:09.827576 3324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:09.829406 3309 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:07:09.829595 3309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "2cdf78aff1bf4d5795d1128d3942c8cf"
format_stamp: "Formatted at 2026-09-25 15:07:09 on dist-test-slave-nsrc"
server_key: "dc9c443f52d7a02c672729c1dd83ea89"
server_key_iv: "4df3f97d771019f7b77b8ef13882b3fe"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:09.829844 3309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:09.867659 3309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:09.872450 3309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:09.872797 3309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:09.891202 3375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:33217 every 8 connection(s)
I20260925 15:07:09.891212 3309 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:33217
I20260925 15:07:09.892935 3309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:09.896715 3376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:09.902878 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 3309
I20260925 15:07:09.903086 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:07:09.903936 23472 external_mini_cluster.cc:1428] Setting key f6b66e1578fd8a064d0d03ebf7a9c0a3
I20260925 15:07:09.908130 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf: Bootstrap starting.
I20260925 15:07:09.912323 3376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:09.913790 3376 log.cc:824] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:09.923256 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf: No bootstrap required, opened a new log
I20260925 15:07:09.931944 3376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33217 } }
I20260925 15:07:09.932407 3376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:09.932512 3376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cdf78aff1bf4d5795d1128d3942c8cf, State: Initialized, Role: FOLLOWER
I20260925 15:07:09.933207 3376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33217 } }
I20260925 15:07:09.933472 3376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:09.933609 3376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:09.933777 3376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 0 FOLLOWER]: Advancing to term 1
Sep 25 15:07:09 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:09.936523 3376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33217 } }
I20260925 15:07:09.937036 3376 leader_election.cc:302] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cdf78aff1bf4d5795d1128d3942c8cf; no voters:
I20260925 15:07:09.938553 3376 leader_election.cc:288] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:09.939231 3382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:09.940819 3376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:09.945746 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:09.906092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.22.236.62:33217)
0925 15:07:09.920675 (+ 14583us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:09.920693 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:09.921129 (+ 436us) server_negotiation.cc:407] Connection header received
0925 15:07:09.923657 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:09.923690 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:09.924243 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:09.924671 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:09.926259 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:09.928354 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:09.930224 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:09.931313 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:09.931485 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:09.936441 (+ 4956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:09.936452 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:09.936473 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:09.936523 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:09.940323 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:09.941189 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:09.941198 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:09.941207 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:09.941386 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:09.941801 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:09.941806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:09.941810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:09.942490 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:09.942767 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:09.943991 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0925 15:07:09.944417 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14338,"thread_start_us":14091,"threads_started":1}
I20260925 15:07:09.949709 3382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 1 LEADER]: Becoming Leader. State: Replica: 2cdf78aff1bf4d5795d1128d3942c8cf, State: Running, Role: LEADER
I20260925 15:07:09.950294 3382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33217 } }
I20260925 15:07:09.953112 3381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33217 } } }
I20260925 15:07:09.953188 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cdf78aff1bf4d5795d1128d3942c8cf. Latest consensus state: current_term: 1 leader_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33217 } } }
I20260925 15:07:09.953557 3381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:09.953559 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:09.954413 3389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:09.969897 3389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:09.983408 3389 catalog_manager.cc:1347] Generated new cluster ID: db53f56956ed4dc1a55b1aa85ecdd9a4
I20260925 15:07:09.983639 3389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:10.003654 3389 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:10.003696 3376 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:07:10.009114 3389 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:10.025730 3389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf: Generated new TSK 0
I20260925 15:07:10.026916 3389 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:07:10.768384 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.236.20:34561/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33217
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:07:11 dist-test-slave-nsrc krb5kdc[2865](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348831, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 15:07:11.190297 3421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:11.190694 3421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:11.190894 3421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:11.202229 3421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:11.202445 3421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:11.202672 3421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:07:11.218019 3421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.236.20:34561/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:11.219760 3421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:11.221551 3421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:11.238077 3431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:11.239641 3432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:11.247805 3435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:11.250455 3421 server_base.cc:1034] running on GCE node
I20260925 15:07:11.251327 3421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:11.256282 3421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:11.257622 3421 hybrid_clock.cc:648] HybridClock initialized: now 1790348831257506 us; error 105 us; skew 500 ppm
I20260925 15:07:11.258074 3421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:11 dist-test-slave-nsrc krb5kdc[2865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348831, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:11.262384 3421 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:07:11.274818 3421 webserver.cc:460] Webserver started at http://127.22.236.1:41011/ using document root <none> and password file <none>
I20260925 15:07:11.275810 3421 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:11.275990 3421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:11.276373 3421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:07:11 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348831, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:07:11.295696 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "2c0f8dbc4f7c46d889ee23657d132d4d"
format_stamp: "Formatted at 2026-09-25 15:07:11 on dist-test-slave-nsrc"
server_key: "a343a955b6ccf99fdb2943b7b6a7f0e7"
server_key_iv: "e4764207e5ce39fa1deae53cf27a97fd"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:11.296873 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "2c0f8dbc4f7c46d889ee23657d132d4d"
format_stamp: "Formatted at 2026-09-25 15:07:11 on dist-test-slave-nsrc"
server_key: "a343a955b6ccf99fdb2943b7b6a7f0e7"
server_key_iv: "e4764207e5ce39fa1deae53cf27a97fd"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:11.304301 3421 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260925 15:07:11.338707 3441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:11.342087 3421 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.002s sys 0.011s
I20260925 15:07:11.348376 3421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "2c0f8dbc4f7c46d889ee23657d132d4d"
format_stamp: "Formatted at 2026-09-25 15:07:11 on dist-test-slave-nsrc"
server_key: "a343a955b6ccf99fdb2943b7b6a7f0e7"
server_key_iv: "e4764207e5ce39fa1deae53cf27a97fd"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:11.348801 3421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:11.405934 3421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:11.411291 3421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:11.411624 3421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:11.412940 3421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:11.414713 3421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:11.414832 3421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:11.414969 3421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:11.415061 3421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:11.463766 3421 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:41107
I20260925 15:07:11.463783 3553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:41107 every 8 connection(s)
I20260925 15:07:11.465272 3421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:11.473129 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 3421
I20260925 15:07:11.473343 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:07:11.474117 23472 external_mini_cluster.cc:1428] Setting key 8969837f9ce6d3b5f103699d9c8ddacd
Sep 25 15:07:11 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348831, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
Sep 25 15:07:11 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348831, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
I20260925 15:07:11.537233 3558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:11.491720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:43339 (local address 127.22.236.62:33217)
0925 15:07:11.508663 (+ 16943us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:11.508671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:11.511586 (+ 2915us) server_negotiation.cc:407] Connection header received
0925 15:07:11.511649 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:11.511652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:11.511737 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:11.511832 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:11.513947 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:11.515092 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:11.520237 (+ 5145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:11.520900 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:11.521033 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:11.528861 (+ 7828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:11.528869 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:11.528874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:11.528920 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:11.532473 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:11.533539 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:11.533546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:11.533549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:11.533719 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:11.534378 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:11.534389 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:11.534393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:11.534732 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:11.534877 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:11.536627 (+ 1750us) server_negotiation.cc:299] Negotiation successful
0925 15:07:11.536854 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16788,"thread_start_us":16635,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
I20260925 15:07:11.538565 3557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:11.491019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33217 (local address 127.22.236.1:43339)
0925 15:07:11.509465 (+ 18446us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:11.509518 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:11.511269 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:11.512187 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:11.512200 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:11.512873 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:11.513715 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:11.513744 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:11.515863 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:11.515869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:11.517802 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:11.517818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:11.521226 (+ 3408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:11.521235 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:11.521564 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:11.522438 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:11.522474 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:11.528506 (+ 6032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:11.532704 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:11.532715 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:11.532739 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:11.533384 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:11.533903 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:11.533909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:11.533912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:11.534217 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:11.535559 (+ 1342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:11.535568 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:11.536104 (+ 536us) client_negotiation.cc:769] Sending connection context
0925 15:07:11.536471 (+ 367us) client_negotiation.cc:240] Negotiation successful
0925 15:07:11.537081 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18011,"thread_start_us":17706,"threads_started":1}
I20260925 15:07:11.541975 3554 heartbeater.cc:347] Connected to a master server at 127.22.236.62:33217
I20260925 15:07:11.542339 3554 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:11.543260 3554 heartbeater.cc:511] Master 127.22.236.62:33217 requested a full tablet report, sending...
I20260925 15:07:11.546236 3341 ts_manager.cc:194] Registered new tserver with Master: 2c0f8dbc4f7c46d889ee23657d132d4d (127.22.236.1:41107)
I20260925 15:07:11.548368 3341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:43339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:07:11.573508 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.236.20:34561/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33217
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:07:11 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348831, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
W20260925 15:07:12.010452 3560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:12.010818 3560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:12.011019 3560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:12.022194 3560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:12.022385 3560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:12.022588 3560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:07:12.037413 3560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.236.20:34561/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:12.039002 3560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:12.040577 3560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:12.056216 3574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:12.056236 3575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:12.057085 3577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:12.057261 3560 server_base.cc:1034] running on GCE node
I20260925 15:07:12.057869 3560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:12.058859 3560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:12.060062 3560 hybrid_clock.cc:648] HybridClock initialized: now 1790348832060004 us; error 48 us; skew 500 ppm
I20260925 15:07:12.060395 3560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:12 dist-test-slave-nsrc krb5kdc[2865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348832, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:12.063413 3560 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:07:12.064931 3560 webserver.cc:460] Webserver started at http://127.22.236.2:44149/ using document root <none> and password file <none>
I20260925 15:07:12.065588 3560 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:12.065707 3560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:12.065999 3560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:07:12 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348832, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:07:12.082921 3560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "c71f5f4472864248a9034703d33e1b3e"
format_stamp: "Formatted at 2026-09-25 15:07:12 on dist-test-slave-nsrc"
server_key: "1bcf79843aef915c6742d4cd2032e76a"
server_key_iv: "3a9befe95065d5464c4a702dad17dcd9"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:12.083971 3560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "c71f5f4472864248a9034703d33e1b3e"
format_stamp: "Formatted at 2026-09-25 15:07:12 on dist-test-slave-nsrc"
server_key: "1bcf79843aef915c6742d4cd2032e76a"
server_key_iv: "3a9befe95065d5464c4a702dad17dcd9"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:12.090653 3560 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 15:07:12.109051 3584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:12.110908 3560 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:07:12.111100 3560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "c71f5f4472864248a9034703d33e1b3e"
format_stamp: "Formatted at 2026-09-25 15:07:12 on dist-test-slave-nsrc"
server_key: "1bcf79843aef915c6742d4cd2032e76a"
server_key_iv: "3a9befe95065d5464c4a702dad17dcd9"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:12.111339 3560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:12.139560 3560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:12.144554 3560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:12.144866 3560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:12.145998 3560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:12.147588 3560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:12.147675 3560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:12.147827 3560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:12.147884 3560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:12.190742 3560 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:44903
I20260925 15:07:12.190795 3696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:44903 every 8 connection(s)
I20260925 15:07:12.192379 3560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:12.194743 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 3560
I20260925 15:07:12.194921 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:07:12.195524 23472 external_mini_cluster.cc:1428] Setting key 31e553ae10c5bb764d68fee70a18cd40
Sep 25 15:07:12 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348832, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
I20260925 15:07:12.229420 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:12.198455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:45107 (local address 127.22.236.62:33217)
0925 15:07:12.208643 (+ 10188us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:12.208650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:12.211856 (+ 3206us) server_negotiation.cc:407] Connection header received
0925 15:07:12.211945 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:12.211950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:12.212047 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:12.212170 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:12.214181 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.215295 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.217630 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.218251 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.218336 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:12.222580 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:12.222587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:12.222592 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:12.222631 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:12.225964 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:12.226839 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:12.226844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:12.226848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:12.227011 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:12.227463 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:12.227467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:12.227470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:12.227790 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:12.227908 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:12.228825 (+ 917us) server_negotiation.cc:299] Negotiation successful
0925 15:07:12.229062 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10056,"thread_start_us":9940,"threads_started":1}
I20260925 15:07:12.231395 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:12.197801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33217 (local address 127.22.236.2:45107)
0925 15:07:12.209226 (+ 11425us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:12.209288 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:12.211475 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:12.212595 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:12.212608 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:12.213193 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:12.213978 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:12.214003 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.215492 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.215497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:12.217474 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:12.217487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.218641 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.218646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:12.218922 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:12.219699 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:12.219730 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:12.222333 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:12.226149 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:12.226158 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:12.226176 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:12.226713 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:12.227139 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:12.227143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:12.227147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:12.227354 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:12.228001 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:12.228009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:12.228427 (+ 418us) client_negotiation.cc:769] Sending connection context
0925 15:07:12.229440 (+ 1013us) client_negotiation.cc:240] Negotiation successful
0925 15:07:12.229974 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11109,"thread_start_us":10860,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
I20260925 15:07:12.233119 3697 heartbeater.cc:347] Connected to a master server at 127.22.236.62:33217
I20260925 15:07:12.233438 3697 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:12.234232 3697 heartbeater.cc:511] Master 127.22.236.62:33217 requested a full tablet report, sending...
I20260925 15:07:12.235747 3341 ts_manager.cc:194] Registered new tserver with Master: c71f5f4472864248a9034703d33e1b3e (127.22.236.2:44903)
I20260925 15:07:12.236690 3341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:45107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:07:12.268775 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.236.20:34561/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33217
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:07:12.552301 3554 heartbeater.cc:503] Master 127.22.236.62:33217 was elected leader, sending a full tablet report...
W20260925 15:07:12.656487 3703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:12.656782 3703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:12.656900 3703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:12.663980 3703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:12.664103 3703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:12.664264 3703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:07:12.674042 3703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.236.20:34561/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:33217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:12.675338 3703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:12.677117 3703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:12.694303 3709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:12.694329 3708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:12.700918 3711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:12.701519 3703 server_base.cc:1034] running on GCE node
I20260925 15:07:12.702170 3703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:12.703147 3703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:12.704339 3703 hybrid_clock.cc:648] HybridClock initialized: now 1790348832704275 us; error 59 us; skew 500 ppm
I20260925 15:07:12.704648 3703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:12 dist-test-slave-nsrc krb5kdc[2865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348832, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:12.707983 3703 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:07:12.709815 3703 webserver.cc:460] Webserver started at http://127.22.236.3:34393/ using document root <none> and password file <none>
I20260925 15:07:12.710695 3703 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:12.710834 3703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:12.711249 3703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:07:12 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348832, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:07:12.727566 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "b51ab5b963eb48f296fea4ff4be4d185"
format_stamp: "Formatted at 2026-09-25 15:07:12 on dist-test-slave-nsrc"
server_key: "fb694fc783d836bed4bf768d82398079"
server_key_iv: "8b5bda6211b240da3b3af7d6e59157d8"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:12.728530 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "b51ab5b963eb48f296fea4ff4be4d185"
format_stamp: "Formatted at 2026-09-25 15:07:12 on dist-test-slave-nsrc"
server_key: "fb694fc783d836bed4bf768d82398079"
server_key_iv: "8b5bda6211b240da3b3af7d6e59157d8"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:12.733744 3703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:07:12.748939 3718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:12.750469 3703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:12.750629 3703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "b51ab5b963eb48f296fea4ff4be4d185"
format_stamp: "Formatted at 2026-09-25 15:07:12 on dist-test-slave-nsrc"
server_key: "fb694fc783d836bed4bf768d82398079"
server_key_iv: "8b5bda6211b240da3b3af7d6e59157d8"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:12.750835 3703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:12.768988 3703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:12.772712 3703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:12.772977 3703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:12.773880 3703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:12.775141 3703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:12.775213 3703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:12.775334 3703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:12.775386 3703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:12.803128 3703 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:39709
I20260925 15:07:12.803148 3830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:39709 every 8 connection(s)
I20260925 15:07:12.804335 3703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:12.808246 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 3703
I20260925 15:07:12.808421 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:07:12.808998 23472 external_mini_cluster.cc:1428] Setting key d14365eda9f21c94fe955ca7a813aa53
Sep 25 15:07:12 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348832, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:12.822751 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:12.806714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:38197 (local address 127.22.236.62:33217)
0925 15:07:12.807135 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:12.807140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:12.808082 (+ 942us) server_negotiation.cc:407] Connection header received
0925 15:07:12.810367 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:12.810372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:12.810484 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:12.810576 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:12.812134 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.812842 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.814289 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.814710 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.814807 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:12.817955 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:12.817960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:12.817963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:12.817996 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:12.820253 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:12.820847 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:12.820851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:12.820853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:12.820980 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:12.821310 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:12.821314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:12.821316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:12.821565 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:12.821678 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:12.822228 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 15:07:12.822383 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":186,"threads_started":1}
I20260925 15:07:12.823820 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:12.807257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33217 (local address 127.22.236.3:38197)
0925 15:07:12.807903 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:12.807951 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:12.810087 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:12.810838 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:12.810847 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:12.811322 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:12.811964 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:12.811989 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.812954 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.812958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:12.814163 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:12.814174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.814865 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.814869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:12.815077 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:12.815670 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:12.815690 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:12.817718 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:12.820354 (+ 2636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:12.820361 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:12.820373 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:12.820754 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:12.821055 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:12.821058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:12.821073 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:12.821225 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:12.821650 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:12.821655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:12.821914 (+ 259us) client_negotiation.cc:769] Sending connection context
0925 15:07:12.822158 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 15:07:12.822641 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":244,"threads_started":1}
I20260925 15:07:12.825088 3831 heartbeater.cc:347] Connected to a master server at 127.22.236.62:33217
I20260925 15:07:12.825323 3831 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:12.825912 3831 heartbeater.cc:511] Master 127.22.236.62:33217 requested a full tablet report, sending...
I20260925 15:07:12.827135 3341 ts_manager.cc:194] Registered new tserver with Master: b51ab5b963eb48f296fea4ff4be4d185 (127.22.236.3:39709)
I20260925 15:07:12.827864 3341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:38197
I20260925 15:07:12.836279 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:07:12.838059 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 3421
Sep 25 15:07:12 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:07:12.856588 3836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:12.838840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.22.236.1:41107)
0925 15:07:12.839377 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:12.839385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:12.839414 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:07:12.839550 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:12.839557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:12.839752 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:12.839883 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:12.841287 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.842311 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.843753 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.844309 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.844415 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:12.849011 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:12.849026 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:12.849045 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:12.849089 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:12.852962 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:12.853929 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:12.853939 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:12.853942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:12.854148 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:12.854605 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:12.854609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:12.854611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:12.855119 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:12.855411 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:12.855957 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 15:07:12.856239 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":229,"threads_started":1}
I20260925 15:07:12.857996 3549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:12.959950 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 3421
I20260925 15:07:12.972034 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 3560
Sep 25 15:07:12 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:07:12.986563 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:12.972832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.22.236.2:44903)
0925 15:07:12.973328 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:12.973335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:12.973360 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:07:12.973418 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:12.973423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:12.973629 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:12.973754 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:12.974961 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.975909 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.977226 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.977747 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.977838 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:12.981861 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:12.981872 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:12.981885 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:12.981915 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:12.984272 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:12.984906 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:12.984912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:12.984914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:12.985025 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:12.985312 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:12.985315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:12.985317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:12.985611 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:12.985822 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:12.986079 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 15:07:12.986261 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":248,"threads_started":1}
I20260925 15:07:12.987673 3692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:13.067591 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 3560
I20260925 15:07:13.083432 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 3703
Sep 25 15:07:13 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:07:13.100811 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:13.084609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57802 (local address 127.22.236.3:39709)
0925 15:07:13.085289 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:13.085298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:13.085330 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:07:13.085409 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:13.085416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:13.085680 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:13.085833 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:13.087302 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:13.088817 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:13.090669 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:13.091204 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:13.091301 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:13.095819 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:13.095830 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:13.095845 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:13.095876 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:13.098496 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:13.099116 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:13.099123 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:13.099125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:13.099239 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:13.099515 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:13.099518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:13.099520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:13.099825 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:13.100067 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:13.100257 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 15:07:13.100464 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":355,"threads_started":1}
I20260925 15:07:13.102001 3826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:13.182170 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 3703
I20260925 15:07:13.193944 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 3309
I20260925 15:07:13.195209 3371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:13.299239 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 3309
I20260925 15:07:13.318734 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:33217
--webserver_interface=127.22.236.62
--webserver_port=46293
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:33217
--ranger_config_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.236.20:34561/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.236.20:34561/kms with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:13.623343 3843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:13.623647 3843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:13.623790 3843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:13.630877 3843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:13.631021 3843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:13.631103 3843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:13.631183 3843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:13.631248 3843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:13.641008 3843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.236.20:34561/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:33217
--ranger_config_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:33217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=46293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:13.642203 3843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:13.643456 3843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:13.654572 3848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:13.654632 3851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:13.654611 3849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:13.655283 3843 server_base.cc:1034] running on GCE node
I20260925 15:07:13.656070 3843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:13.657507 3843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:13.658687 3843 hybrid_clock.cc:648] HybridClock initialized: now 1790348833658636 us; error 45 us; skew 500 ppm
I20260925 15:07:13.659020 3843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:13 dist-test-slave-nsrc krb5kdc[2865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348833, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:13.661850 3843 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:07:13.663179 3843 webserver.cc:460] Webserver started at http://127.22.236.62:46293/ using document root <none> and password file <none>
I20260925 15:07:13.663774 3843 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:13.663889 3843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:07:13 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348833, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:07:13.711731 3843 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:07:13.715287 3859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:13.716704 3843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 15:07:13.716861 3843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "2cdf78aff1bf4d5795d1128d3942c8cf"
format_stamp: "Formatted at 2026-09-25 15:07:09 on dist-test-slave-nsrc"
server_key: "dc9c443f52d7a02c672729c1dd83ea89"
server_key_iv: "4df3f97d771019f7b77b8ef13882b3fe"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:13.717770 3843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:13.746148 3843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:13.749593 3843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:13.749840 3843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:13.766101 3843 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:33217
I20260925 15:07:13.766124 3910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:33217 every 8 connection(s)
I20260925 15:07:13.767506 3843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:13.768067 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 3843
I20260925 15:07:13.769114 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.236.20:34561/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:41107
--local_ip_for_outbound_sockets=127.22.236.1
--tserver_master_addrs=127.22.236.62:33217
--webserver_port=41011
--webserver_interface=127.22.236.1
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.236.20:34561/kms with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:07:13.774272 3911 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:07:13.777361 3911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf: Bootstrap starting.
I20260925 15:07:13.795149 3911 log.cc:824] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:13.803130 3911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:07:13.803578 3911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf: Bootstrap complete.
I20260925 15:07:13.811916 3911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33217 } }
I20260925 15:07:13.812961 3911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cdf78aff1bf4d5795d1128d3942c8cf, State: Initialized, Role: FOLLOWER
I20260925 15:07:13.813570 3911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33217 } }
I20260925 15:07:13.813784 3911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:13.813886 3911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:13.814045 3911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:07:13.817660 3911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33217 } }
I20260925 15:07:13.818101 3911 leader_election.cc:302] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cdf78aff1bf4d5795d1128d3942c8cf; no voters:
I20260925 15:07:13.818670 3911 leader_election.cc:288] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:07:13.818796 3915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:07:13.819844 3915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [term 2 LEADER]: Becoming Leader. State: Replica: 2cdf78aff1bf4d5795d1128d3942c8cf, State: Running, Role: LEADER
I20260925 15:07:13.820432 3915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33217 } }
I20260925 15:07:13.820796 3911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:13.825909 3916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33217 } } }
I20260925 15:07:13.825917 3917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cdf78aff1bf4d5795d1128d3942c8cf. Latest consensus state: current_term: 2 leader_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf78aff1bf4d5795d1128d3942c8cf" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 33217 } } }
I20260925 15:07:13.826459 3917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:13.826359 3916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:13.837146 3924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:13.842238 3924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:13.843014 3911 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:07:13.843674 3924 catalog_manager.cc:1259] Loaded cluster ID: db53f56956ed4dc1a55b1aa85ecdd9a4
I20260925 15:07:13.844321 3924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:13.848256 3924 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:13.850431 3924 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf: Loaded TSK: 0
I20260925 15:07:13.851990 3924 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:07:14.159157 3913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:14.159716 3913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:14.159963 3913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:14.171936 3913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:14.172175 3913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:14.172432 3913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:07:14.187902 3913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.236.20:34561/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:41107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=41011
--tserver_master_addrs=127.22.236.62:33217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:14.189930 3913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:14.191763 3913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:14.220932 3949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:14.222056 3948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:14.223860 3952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:14.224381 3913 server_base.cc:1034] running on GCE node
I20260925 15:07:14.225342 3913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:14.228443 3913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:14.229734 3913 hybrid_clock.cc:648] HybridClock initialized: now 1790348834229661 us; error 62 us; skew 500 ppm
I20260925 15:07:14.230161 3913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:14 dist-test-slave-nsrc krb5kdc[2865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348834, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:14.234459 3913 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:07:14.237123 3913 webserver.cc:460] Webserver started at http://127.22.236.1:41011/ using document root <none> and password file <none>
I20260925 15:07:14.238098 3913 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:14.238256 3913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:07:14 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348834, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:07:14.294878 3913 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.011s sys 0.000s
I20260925 15:07:14.305531 3959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:14.307770 3913 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.007s
I20260925 15:07:14.308037 3913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "2c0f8dbc4f7c46d889ee23657d132d4d"
format_stamp: "Formatted at 2026-09-25 15:07:11 on dist-test-slave-nsrc"
server_key: "a343a955b6ccf99fdb2943b7b6a7f0e7"
server_key_iv: "e4764207e5ce39fa1deae53cf27a97fd"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:14.309623 3913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:14.362130 3913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:14.367200 3913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:14.367558 3913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:14.382302 3913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:14.395206 3913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:14.395345 3913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:14.395539 3913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:14.395619 3913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:14.463068 3913 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:41107
I20260925 15:07:14.463133 4072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:41107 every 8 connection(s)
I20260925 15:07:14.465059 3913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:14.471638 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 3913
I20260925 15:07:14.472582 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.236.20:34561/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:44903
--local_ip_for_outbound_sockets=127.22.236.2
--tserver_master_addrs=127.22.236.62:33217
--webserver_port=44149
--webserver_interface=127.22.236.2
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.236.20:34561/kms with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:07:14 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348834, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:14.508229 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:14.468398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33217 (local address 127.22.236.1:55469)
0925 15:07:14.482657 (+ 14259us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:14.482721 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:14.486232 (+ 3511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:14.488205 (+ 1973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:14.488218 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:14.489018 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:14.489833 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:14.489873 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:14.492236 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:14.492241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:14.494240 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:14.494256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:14.495721 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:14.495729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:14.496022 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:14.496874 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:14.496908 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:14.499549 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:14.503352 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:14.503361 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:14.503378 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:14.503966 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:14.504561 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:14.504567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:14.504570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:14.504755 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:14.505537 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:14.505544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:14.505911 (+ 367us) client_negotiation.cc:769] Sending connection context
0925 15:07:14.506214 (+ 303us) client_negotiation.cc:240] Negotiation successful
0925 15:07:14.506764 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13852,"thread_start_us":14411,"threads_started":1}
I20260925 15:07:14.510596 4076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:14.468949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:55469 (local address 127.22.236.62:33217)
0925 15:07:14.470915 (+ 1966us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:14.470929 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:14.482896 (+ 11967us) server_negotiation.cc:407] Connection header received
0925 15:07:14.486983 (+ 4087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:14.487003 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:14.487534 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:14.487944 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:14.490090 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:14.491999 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:14.494403 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:14.495462 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:14.495665 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:14.499881 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:14.499893 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:14.499914 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:14.499964 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:14.503177 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:14.504174 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:14.504184 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:14.504193 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:14.504392 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:14.504897 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:14.504903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:14.504905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:14.505403 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:14.505578 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:14.508932 (+ 3354us) server_negotiation.cc:299] Negotiation successful
0925 15:07:14.509357 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1740,"thread_start_us":1512,"threads_started":1}
I20260925 15:07:14.514549 4073 heartbeater.cc:347] Connected to a master server at 127.22.236.62:33217
I20260925 15:07:14.514885 4073 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:14.515831 4073 heartbeater.cc:511] Master 127.22.236.62:33217 requested a full tablet report, sending...
W20260925 15:07:14.518676 4073 heartbeater.cc:644] Failed to heartbeat to 127.22.236.62:33217 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:07:14.518925 4073 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:14.519233 4073 heartbeater.cc:511] Master 127.22.236.62:33217 requested a full tablet report, sending...
I20260925 15:07:14.520371 4073 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:14.520654 4073 heartbeater.cc:511] Master 127.22.236.62:33217 requested a full tablet report, sending...
W20260925 15:07:14.521660 4073 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:07:14.856405 4077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:14.856757 4077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:14.856979 4077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:14.868119 4077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:14.868314 4077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:14.868501 4077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:07:14.883683 4077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.236.20:34561/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:44903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=44149
--tserver_master_addrs=127.22.236.62:33217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:14.885435 4077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:14.887137 4077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:14.901414 4091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:14.901388 4092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:14.902256 4077 server_base.cc:1034] running on GCE node
W20260925 15:07:14.902313 4094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:14.903060 4077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:14.904476 4077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:14.905741 4077 hybrid_clock.cc:648] HybridClock initialized: now 1790348834905642 us; error 85 us; skew 500 ppm
I20260925 15:07:14.906111 4077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:14 dist-test-slave-nsrc krb5kdc[2865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348834, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:14.915004 4077 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:07:14.941385 4077 webserver.cc:460] Webserver started at http://127.22.236.2:44149/ using document root <none> and password file <none>
I20260925 15:07:14.942099 4077 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:14.942235 4077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:07:14 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348834, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:07:14.976091 4077 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 15:07:14.979638 4104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:14.981422 4077 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:07:14.981588 4077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "c71f5f4472864248a9034703d33e1b3e"
format_stamp: "Formatted at 2026-09-25 15:07:12 on dist-test-slave-nsrc"
server_key: "1bcf79843aef915c6742d4cd2032e76a"
server_key_iv: "3a9befe95065d5464c4a702dad17dcd9"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:14.982944 4077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:15.000062 4077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:15.005206 4077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:15.005507 4077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:15.006657 4077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:15.008517 4077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:15.008664 4077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:15.008793 4077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:15.008893 4077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:07:15 dist-test-slave-nsrc krb5kdc[2865](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348835, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:15.090171 4077 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:44903
I20260925 15:07:15.090368 4217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:44903 every 8 connection(s)
I20260925 15:07:15.092193 4077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:15.092739 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 4077
I20260925 15:07:15.093535 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.236.20:34561/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:39709
--local_ip_for_outbound_sockets=127.22.236.3
--tserver_master_addrs=127.22.236.62:33217
--webserver_port=34393
--webserver_interface=127.22.236.3
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.236.20:34561/kms with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:07:15 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348834, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:15.120873 4221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:15.096708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:38147 (local address 127.22.236.62:33217)
0925 15:07:15.100968 (+ 4260us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:15.100973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:15.100993 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:07:15.103300 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:15.103306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:15.103397 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:15.103512 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:15.105622 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.106706 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.108910 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.109548 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.109637 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:15.113708 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:15.113714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:15.113718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:15.113755 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:15.117169 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:15.118079 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:15.118085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:15.118088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:15.118234 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:15.118750 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:15.118756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:15.118759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:15.119111 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:15.119253 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:15.120232 (+ 979us) server_negotiation.cc:299] Negotiation successful
0925 15:07:15.120505 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4136,"thread_start_us":4016,"threads_started":1}
I20260925 15:07:15.123423 4220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:15.096010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33217 (local address 127.22.236.2:38147)
0925 15:07:15.098374 (+ 2364us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:15.098429 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:15.103023 (+ 4594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:15.103945 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:15.103957 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:15.104626 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:15.105414 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:15.105439 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.106885 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.106890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:15.108754 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:15.108769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.109987 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.109994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:15.110264 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:15.111036 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:15.111068 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:15.113460 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:15.117330 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:15.117339 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:15.117357 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:15.117925 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:15.118364 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:15.118369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:15.118373 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:15.118613 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:15.119325 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:15.119334 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:15.119772 (+ 438us) client_negotiation.cc:769] Sending connection context
0925 15:07:15.121346 (+ 1574us) client_negotiation.cc:240] Negotiation successful
0925 15:07:15.121958 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2075,"thread_start_us":1819,"threads_started":1}
I20260925 15:07:15.125259 4218 heartbeater.cc:347] Connected to a master server at 127.22.236.62:33217
I20260925 15:07:15.125583 4218 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:15.126363 4218 heartbeater.cc:511] Master 127.22.236.62:33217 requested a full tablet report, sending...
I20260925 15:07:15.128335 3876 ts_manager.cc:194] Registered new tserver with Master: c71f5f4472864248a9034703d33e1b3e (127.22.236.2:44903)
I20260925 15:07:15.129894 3876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:38147
Sep 25 15:07:15 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348835, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
W20260925 15:07:15.484633 4222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:15.484939 4222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:15.485077 4222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:15.495208 4222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:15.495404 4222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:15.495611 4222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:07:15.509449 4222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.236.20:34561/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:39709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=34393
--tserver_master_addrs=127.22.236.62:33217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:15.511312 4222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:15.513070 4222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:07:15.521811 4073 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:15.522228 4073 heartbeater.cc:511] Master 127.22.236.62:33217 requested a full tablet report, sending...
I20260925 15:07:15.523547 3876 ts_manager.cc:194] Registered new tserver with Master: 2c0f8dbc4f7c46d889ee23657d132d4d (127.22.236.1:41107)
I20260925 15:07:15.524814 3876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:55469
W20260925 15:07:15.556461 4227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:15.565831 4228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:15.575250 4222 server_base.cc:1034] running on GCE node
W20260925 15:07:15.575357 4230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:15.576112 4222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:15.577201 4222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:15.578455 4222 hybrid_clock.cc:648] HybridClock initialized: now 1790348835578394 us; error 59 us; skew 500 ppm
I20260925 15:07:15.578765 4222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:15 dist-test-slave-nsrc krb5kdc[2865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348835, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:15.581794 4222 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:07:15.583542 4222 webserver.cc:460] Webserver started at http://127.22.236.3:34393/ using document root <none> and password file <none>
I20260925 15:07:15.584277 4222 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:15.584405 4222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:07:15 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348835, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:07:15.630296 4222 fs_manager.cc:709] Time spent opening directory manager: real 0.029s user 0.009s sys 0.018s
I20260925 15:07:15.648923 4238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:15.653716 4222 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.013s sys 0.004s
I20260925 15:07:15.653937 4222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "b51ab5b963eb48f296fea4ff4be4d185"
format_stamp: "Formatted at 2026-09-25 15:07:12 on dist-test-slave-nsrc"
server_key: "fb694fc783d836bed4bf768d82398079"
server_key_iv: "8b5bda6211b240da3b3af7d6e59157d8"
server_key_version: "kuduclusterkey@0"
I20260925 15:07:15.655448 4222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:15.706810 4222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:15.710659 4222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:15.710925 4222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:15.711951 4222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:15.713382 4222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:15.713483 4222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:15.713614 4222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:15.713683 4222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:15.755177 4222 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:39709
I20260925 15:07:15.755259 4350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:39709 every 8 connection(s)
I20260925 15:07:15.757056 4222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:15.772670 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 4222
Sep 25 15:07:15 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348835, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:15.825058 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:15.797611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.22.236.62:33217)
0925 15:07:15.803973 (+ 6362us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:15.803979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:15.803996 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:07:15.804063 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:15.804069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:15.804260 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:15.806701 (+ 2441us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:15.806867 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.807996 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.812769 (+ 4773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.813191 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.813302 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:15.816867 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:15.816873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:15.816878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:15.816915 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:15.820695 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:15.822335 (+ 1640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:15.822341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:15.822344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:15.822497 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:15.822936 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:15.822940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:15.822943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:15.823301 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:15.823573 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:15.824382 (+ 809us) server_negotiation.cc:299] Negotiation successful
0925 15:07:15.824654 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6204,"thread_start_us":6541,"threads_started":1}
I20260925 15:07:15.830013 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:15.761030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:49725 (local address 127.22.236.62:33217)
0925 15:07:15.797399 (+ 36369us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:15.797407 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:15.800605 (+ 3198us) server_negotiation.cc:407] Connection header received
0925 15:07:15.810138 (+ 9533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:15.810145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:15.810246 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:15.810321 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:15.814296 (+ 3975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.815595 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.818060 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.818748 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.818839 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:15.823003 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:15.823009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:15.823013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:15.823059 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:15.826365 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:15.827258 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:15.827263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:15.827266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:15.827400 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:15.827926 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:15.827932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:15.827935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:15.828303 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:15.828432 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:15.829468 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0925 15:07:15.829674 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36218,"thread_start_us":36392,"threads_started":1}
I20260925 15:07:15.831301 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:15.760422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:33217 (local address 127.22.236.3:49725)
0925 15:07:15.799272 (+ 38850us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:15.799344 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:15.803223 (+ 3879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:15.811019 (+ 7796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:15.811035 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:15.811764 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:15.813997 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:15.814024 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.815804 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.815809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:15.817893 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:15.817907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.819225 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.819232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:15.819514 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:15.820207 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:15.820238 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:15.822716 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:15.826524 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:15.826534 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:15.826553 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:15.827124 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:15.827544 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:15.827548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:15.827551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:15.827785 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:15.828521 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:15.828529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:15.828998 (+ 469us) client_negotiation.cc:769] Sending connection context
0925 15:07:15.829347 (+ 349us) client_negotiation.cc:240] Negotiation successful
0925 15:07:15.829891 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38445,"thread_start_us":40101,"threads_started":1}
I20260925 15:07:15.832870 4351 heartbeater.cc:347] Connected to a master server at 127.22.236.62:33217
I20260925 15:07:15.833191 4351 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:15.834034 4351 heartbeater.cc:511] Master 127.22.236.62:33217 requested a full tablet report, sending...
I20260925 15:07:15.835500 3876 ts_manager.cc:194] Registered new tserver with Master: b51ab5b963eb48f296fea4ff4be4d185 (127.22.236.3:39709)
I20260925 15:07:15.836536 3876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:49725
I20260925 15:07:15.840343 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:07:15 dist-test-slave-nsrc krb5kdc[2865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348835, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for HTTP/127.22.236.20@KRBTEST.COM
I20260925 15:07:15.871894 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:15.845801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.22.236.62:33217)
0925 15:07:15.845976 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:15.845982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:15.855301 (+ 9319us) server_negotiation.cc:407] Connection header received
0925 15:07:15.855393 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:15.855398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:15.855495 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:15.855601 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:15.857517 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.858763 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.860969 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.861671 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.861771 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:15.865265 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:15.865272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:15.865277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:15.865318 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:15.868595 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:15.869502 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:15.869508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:15.869512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:15.869667 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:15.870118 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:15.870123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:15.870125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:15.870511 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:15.870723 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:15.871328 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 15:07:15.871565 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:07:15.878717 3876 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:15.895267 3876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:15.956918 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:15.923675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:39709 (local address 127.0.0.1:57818)
0925 15:07:15.943271 (+ 19596us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:15.943322 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:15.943549 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:15.945083 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:15.945090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:15.945120 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:15.945570 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:15.945586 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.951943 (+ 6357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.951949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:15.953960 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:15.953972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.955958 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.955965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:15.956170 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:15.956225 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:07:15.956353 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 15:07:15.956558 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19328,"thread_start_us":23391,"threads_started":1}
I20260925 15:07:15.957670 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:15.929082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:41107 (local address 127.0.0.1:45552)
0925 15:07:15.944362 (+ 15280us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:15.944380 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:15.944498 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:15.945802 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:15.945808 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:15.945834 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:15.946229 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:15.946241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.950357 (+ 4116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.950363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:15.953755 (+ 3392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:15.953770 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.956938 (+ 3168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.956943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:15.957114 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:15.957142 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:07:15.957211 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:07:15.957328 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14751,"mutex_wait_us":1,"thread_start_us":16808,"threads_started":1}
I20260925 15:07:15.958197 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:15.924220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.22.236.3:39709)
0925 15:07:15.942816 (+ 18596us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:15.942825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:15.944273 (+ 1448us) server_negotiation.cc:407] Connection header received
0925 15:07:15.944386 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:15.944394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:15.944673 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:15.944896 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:15.947528 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.951784 (+ 4256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.954119 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.955451 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.955585 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:15.955818 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:15.957701 (+ 1883us) server_negotiation.cc:299] Negotiation successful
0925 15:07:15.957861 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18421,"thread_start_us":18274,"threads_started":1}
I20260925 15:07:15.961382 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:15.929666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.22.236.1:41107)
0925 15:07:15.944208 (+ 14542us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:15.944217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:15.945039 (+ 822us) server_negotiation.cc:407] Connection header received
0925 15:07:15.945134 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:15.945141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:15.945364 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:15.945549 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:15.947535 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.950213 (+ 2678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.953989 (+ 3776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.955451 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.955587 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:15.959832 (+ 4245us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:15.960888 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0925 15:07:15.961081 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14343,"thread_start_us":14188,"threads_started":1}
I20260925 15:07:15.962852 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:15.929232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:44903 (local address 127.0.0.1:60194)
0925 15:07:15.946162 (+ 16930us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:15.946181 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:15.946295 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:15.948220 (+ 1925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:15.948225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:15.948250 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:15.948582 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:15.948594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.950248 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.950253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:15.959624 (+ 9371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:15.959637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.962088 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.962096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:15.962266 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:15.962297 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:07:15.962371 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:07:15.962504 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16794,"mutex_wait_us":7329,"thread_start_us":13621,"threads_started":1}
I20260925 15:07:15.962314 4286 tablet_service.cc:1467] Processing CreateTablet for tablet 9372df41ed0b46bebc4dac825dcd0cd9 (DEFAULT_TABLE table=test-table [id=46f74c68b30641fd9a99cee1b85bbdc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:15.963550 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:15.929980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60194 (local address 127.22.236.2:44903)
0925 15:07:15.944979 (+ 14999us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:15.944988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:15.946407 (+ 1419us) server_negotiation.cc:407] Connection header received
0925 15:07:15.946518 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:15.946525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:15.946742 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:15.946910 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:15.948751 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.950095 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.959786 (+ 9691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:15.961518 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:15.961649 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:15.961907 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:15.963071 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0925 15:07:15.963237 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14809,"thread_start_us":14689,"threads_started":1}
I20260925 15:07:15.963899 4286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9372df41ed0b46bebc4dac825dcd0cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:15.964892 4008 tablet_service.cc:1467] Processing CreateTablet for tablet 9372df41ed0b46bebc4dac825dcd0cd9 (DEFAULT_TABLE table=test-table [id=46f74c68b30641fd9a99cee1b85bbdc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:15.965945 4008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9372df41ed0b46bebc4dac825dcd0cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:15.971621 4152 tablet_service.cc:1467] Processing CreateTablet for tablet 9372df41ed0b46bebc4dac825dcd0cd9 (DEFAULT_TABLE table=test-table [id=46f74c68b30641fd9a99cee1b85bbdc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:15.972946 4152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9372df41ed0b46bebc4dac825dcd0cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:15.989993 4374 tablet_bootstrap.cc:492] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d: Bootstrap starting.
I20260925 15:07:15.992758 4376 tablet_bootstrap.cc:492] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e: Bootstrap starting.
I20260925 15:07:15.994203 4375 tablet_bootstrap.cc:492] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185: Bootstrap starting.
I20260925 15:07:15.995695 4374 tablet_bootstrap.cc:654] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:15.996169 4376 tablet_bootstrap.cc:654] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:15.997151 4374 log.cc:824] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:15.997277 4376 log.cc:824] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:15.999212 4375 tablet_bootstrap.cc:654] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:16.000579 4375 log.cc:824] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:16.005587 4375 tablet_bootstrap.cc:492] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185: No bootstrap required, opened a new log
I20260925 15:07:16.005940 4375 ts_tablet_manager.cc:1397] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260925 15:07:16.015206 4376 tablet_bootstrap.cc:492] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e: No bootstrap required, opened a new log
I20260925 15:07:16.015616 4376 ts_tablet_manager.cc:1397] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.016s
I20260925 15:07:16.018285 4374 tablet_bootstrap.cc:492] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d: No bootstrap required, opened a new log
I20260925 15:07:16.018615 4374 ts_tablet_manager.cc:1397] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.015s
I20260925 15:07:16.028813 4376 raft_consensus.cc:348] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ab5b963eb48f296fea4ff4be4d185" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 39709 } } peers { permanent_uuid: "2c0f8dbc4f7c46d889ee23657d132d4d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41107 } } peers { permanent_uuid: "c71f5f4472864248a9034703d33e1b3e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44903 } }
I20260925 15:07:16.029217 4374 raft_consensus.cc:348] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ab5b963eb48f296fea4ff4be4d185" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 39709 } } peers { permanent_uuid: "2c0f8dbc4f7c46d889ee23657d132d4d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41107 } } peers { permanent_uuid: "c71f5f4472864248a9034703d33e1b3e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44903 } }
I20260925 15:07:16.029736 4374 raft_consensus.cc:374] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:16.029872 4374 raft_consensus.cc:729] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c0f8dbc4f7c46d889ee23657d132d4d, State: Initialized, Role: FOLLOWER
I20260925 15:07:16.031260 4375 raft_consensus.cc:348] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ab5b963eb48f296fea4ff4be4d185" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 39709 } } peers { permanent_uuid: "2c0f8dbc4f7c46d889ee23657d132d4d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41107 } } peers { permanent_uuid: "c71f5f4472864248a9034703d33e1b3e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44903 } }
I20260925 15:07:16.031771 4375 raft_consensus.cc:374] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:16.031909 4375 raft_consensus.cc:729] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51ab5b963eb48f296fea4ff4be4d185, State: Initialized, Role: FOLLOWER
I20260925 15:07:16.032087 4374 consensus_queue.cc:260] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ab5b963eb48f296fea4ff4be4d185" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 39709 } } peers { permanent_uuid: "2c0f8dbc4f7c46d889ee23657d132d4d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41107 } } peers { permanent_uuid: "c71f5f4472864248a9034703d33e1b3e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44903 } }
I20260925 15:07:16.032645 4375 consensus_queue.cc:260] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ab5b963eb48f296fea4ff4be4d185" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 39709 } } peers { permanent_uuid: "2c0f8dbc4f7c46d889ee23657d132d4d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41107 } } peers { permanent_uuid: "c71f5f4472864248a9034703d33e1b3e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44903 } }
I20260925 15:07:16.033897 4376 raft_consensus.cc:374] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:16.034226 4376 raft_consensus.cc:729] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71f5f4472864248a9034703d33e1b3e, State: Initialized, Role: FOLLOWER
I20260925 15:07:16.035281 4376 consensus_queue.cc:260] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ab5b963eb48f296fea4ff4be4d185" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 39709 } } peers { permanent_uuid: "2c0f8dbc4f7c46d889ee23657d132d4d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41107 } } peers { permanent_uuid: "c71f5f4472864248a9034703d33e1b3e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44903 } }
I20260925 15:07:16.045089 4351 heartbeater.cc:503] Master 127.22.236.62:33217 was elected leader, sending a full tablet report...
I20260925 15:07:16.048868 4073 heartbeater.cc:503] Master 127.22.236.62:33217 was elected leader, sending a full tablet report...
I20260925 15:07:16.049619 4374 ts_tablet_manager.cc:1428] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d: Time spent starting tablet: real 0.031s user 0.026s sys 0.001s
I20260925 15:07:16.051038 4218 heartbeater.cc:503] Master 127.22.236.62:33217 was elected leader, sending a full tablet report...
I20260925 15:07:16.051661 4376 ts_tablet_manager.cc:1428] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e: Time spent starting tablet: real 0.036s user 0.011s sys 0.003s
I20260925 15:07:16.052475 4375 ts_tablet_manager.cc:1428] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185: Time spent starting tablet: real 0.046s user 0.014s sys 0.009s
W20260925 15:07:16.063035 4352 tablet.cc:2369] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:07:16.108753 4219 tablet.cc:2369] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:07:16.217707 4074 tablet.cc:2369] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:07:16.326885 4382 raft_consensus.cc:482] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:16.327224 4382 raft_consensus.cc:504] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ab5b963eb48f296fea4ff4be4d185" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 39709 } } peers { permanent_uuid: "2c0f8dbc4f7c46d889ee23657d132d4d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41107 } } peers { permanent_uuid: "c71f5f4472864248a9034703d33e1b3e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44903 } }
I20260925 15:07:16.329126 4382 leader_election.cc:288] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b51ab5b963eb48f296fea4ff4be4d185 (127.22.236.3:39709), 2c0f8dbc4f7c46d889ee23657d132d4d (127.22.236.1:41107)
I20260925 15:07:16.336104 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:16.330576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:34109 (local address 127.22.236.3:39709)
0925 15:07:16.330749 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:16.330755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:16.330776 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:07:16.330856 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:16.330862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:16.330951 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:16.331041 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:16.331738 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.332668 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.334548 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.335275 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.335416 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:16.335488 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:16.335746 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 15:07:16.335862 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:07:16.336162 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:16.329455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:39709 (local address 127.22.236.2:34109)
0925 15:07:16.330552 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:16.330573 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:16.330705 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:16.331172 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:16.331177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:16.331206 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:16.331579 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:16.331592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.332841 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.332847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:16.334400 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:16.334420 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.335473 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.335478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:16.335608 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:16.335632 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 15:07:16.335710 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:07:16.335827 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":780,"threads_started":1}
I20260925 15:07:16.337435 4391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:16.329780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:41107 (local address 127.22.236.2:48197)
0925 15:07:16.331198 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:16.331214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:16.331350 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:16.331690 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:16.331694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:16.331718 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:16.332032 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:16.332042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.333316 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.333321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:16.335345 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:16.335359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.336875 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.336882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:16.337012 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:16.337035 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:07:16.337093 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 15:07:16.337174 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1305,"thread_start_us":1131,"threads_started":1}
I20260925 15:07:16.337357 4306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9372df41ed0b46bebc4dac825dcd0cd9" candidate_uuid: "c71f5f4472864248a9034703d33e1b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51ab5b963eb48f296fea4ff4be4d185" is_pre_election: true
I20260925 15:07:16.337847 4306 raft_consensus.cc:2454] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c71f5f4472864248a9034703d33e1b3e in term 0.
I20260925 15:07:16.338182 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:16.330172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:48197 (local address 127.22.236.1:41107)
0925 15:07:16.330373 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:16.330378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:16.331304 (+ 926us) server_negotiation.cc:407] Connection header received
0925 15:07:16.331494 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:16.331500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:16.331576 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:16.331666 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:16.332203 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.333177 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.335529 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.336682 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.336832 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:16.337830 (+ 998us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:16.337872 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 15:07:16.337969 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:07:16.338649 4108 leader_election.cc:302] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b51ab5b963eb48f296fea4ff4be4d185, c71f5f4472864248a9034703d33e1b3e; no voters:
I20260925 15:07:16.339057 4382 raft_consensus.cc:2781] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:16.338970 4028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9372df41ed0b46bebc4dac825dcd0cd9" candidate_uuid: "c71f5f4472864248a9034703d33e1b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0f8dbc4f7c46d889ee23657d132d4d" is_pre_election: true
I20260925 15:07:16.339210 4382 raft_consensus.cc:482] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:16.339293 4382 raft_consensus.cc:3037] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:16.339496 4028 raft_consensus.cc:2454] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c71f5f4472864248a9034703d33e1b3e in term 0.
I20260925 15:07:16.341849 4382 raft_consensus.cc:504] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ab5b963eb48f296fea4ff4be4d185" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 39709 } } peers { permanent_uuid: "2c0f8dbc4f7c46d889ee23657d132d4d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41107 } } peers { permanent_uuid: "c71f5f4472864248a9034703d33e1b3e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44903 } }
I20260925 15:07:16.342377 4382 leader_election.cc:288] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [CANDIDATE]: Term 1 election: Requested vote from peers b51ab5b963eb48f296fea4ff4be4d185 (127.22.236.3:39709), 2c0f8dbc4f7c46d889ee23657d132d4d (127.22.236.1:41107)
I20260925 15:07:16.342837 4306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9372df41ed0b46bebc4dac825dcd0cd9" candidate_uuid: "c71f5f4472864248a9034703d33e1b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51ab5b963eb48f296fea4ff4be4d185"
I20260925 15:07:16.342861 4028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9372df41ed0b46bebc4dac825dcd0cd9" candidate_uuid: "c71f5f4472864248a9034703d33e1b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0f8dbc4f7c46d889ee23657d132d4d"
I20260925 15:07:16.343019 4306 raft_consensus.cc:3037] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:16.343066 4028 raft_consensus.cc:3037] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:16.345170 4306 raft_consensus.cc:2454] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c71f5f4472864248a9034703d33e1b3e in term 1.
I20260925 15:07:16.345517 4108 leader_election.cc:302] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b51ab5b963eb48f296fea4ff4be4d185, c71f5f4472864248a9034703d33e1b3e; no voters:
I20260925 15:07:16.345755 4382 raft_consensus.cc:2781] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:16.346132 4028 raft_consensus.cc:2454] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c71f5f4472864248a9034703d33e1b3e in term 1.
I20260925 15:07:16.346169 4382 raft_consensus.cc:686] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [term 1 LEADER]: Becoming Leader. State: Replica: c71f5f4472864248a9034703d33e1b3e, State: Running, Role: LEADER
I20260925 15:07:16.346633 4382 consensus_queue.cc:237] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ab5b963eb48f296fea4ff4be4d185" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 39709 } } peers { permanent_uuid: "2c0f8dbc4f7c46d889ee23657d132d4d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41107 } } peers { permanent_uuid: "c71f5f4472864248a9034703d33e1b3e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44903 } }
I20260925 15:07:16.351336 3876 catalog_manager.cc:5495] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e reported cstate change: term changed from 0 to 1, leader changed from <none> to c71f5f4472864248a9034703d33e1b3e (127.22.236.2). New cstate: current_term: 1 leader_uuid: "c71f5f4472864248a9034703d33e1b3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b51ab5b963eb48f296fea4ff4be4d185" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 39709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c0f8dbc4f7c46d889ee23657d132d4d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c71f5f4472864248a9034703d33e1b3e" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44903 } health_report { overall_health: HEALTHY } } }
I20260925 15:07:16.417621 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:16.412198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.22.236.2:44903)
0925 15:07:16.412422 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:16.412428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:16.412665 (+ 237us) server_negotiation.cc:407] Connection header received
0925 15:07:16.412858 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:16.412861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:16.412977 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:16.413098 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:16.413509 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.414221 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.415588 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.416049 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.416164 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:16.416602 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:16.417136 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:16.417245 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:16.417287 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 15:07:16.417352 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:07:16.425802 4028 raft_consensus.cc:1261] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d [term 1 FOLLOWER]: Refusing update from remote peer c71f5f4472864248a9034703d33e1b3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:16.425844 4306 raft_consensus.cc:1261] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185 [term 1 FOLLOWER]: Refusing update from remote peer c71f5f4472864248a9034703d33e1b3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:16.426733 4392 consensus_queue.cc:1035] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c0f8dbc4f7c46d889ee23657d132d4d" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 41107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:16.427068 4382 consensus_queue.cc:1035] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b51ab5b963eb48f296fea4ff4be4d185" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 39709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:16.439100 4397 mvcc.cc:205] Tried to move back new op lower bound from 7333268833989910528 to 7333268833679732736. Current Snapshot: MvccSnapshot[applied={T|T < 7333268833989910528}]
I20260925 15:07:16.443579 4398 mvcc.cc:205] Tried to move back new op lower bound from 7333268833989910528 to 7333268833679732736. Current Snapshot: MvccSnapshot[applied={T|T < 7333268833989910528}]
I20260925 15:07:16.456420 3875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:16.456640 3875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:16.459669 3875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cdf78aff1bf4d5795d1128d3942c8cf: Sending DeleteTablet for 3 replicas of tablet 9372df41ed0b46bebc4dac825dcd0cd9
I20260925 15:07:16.460795 4008 tablet_service.cc:1514] Processing DeleteTablet for tablet 9372df41ed0b46bebc4dac825dcd0cd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:16 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:45552
I20260925 15:07:16.460984 4286 tablet_service.cc:1514] Processing DeleteTablet for tablet 9372df41ed0b46bebc4dac825dcd0cd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:16 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:57818
I20260925 15:07:16.461702 4406 tablet_replica.cc:330] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d: stopping tablet replica
I20260925 15:07:16.461632 4152 tablet_service.cc:1514] Processing DeleteTablet for tablet 9372df41ed0b46bebc4dac825dcd0cd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:16 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:60194
I20260925 15:07:16.462164 4406 raft_consensus.cc:2229] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:16.462736 4406 raft_consensus.cc:2258] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:16.463668 4408 tablet_replica.cc:330] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e: stopping tablet replica
I20260925 15:07:16.464023 4408 raft_consensus.cc:2229] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:16.464538 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 3913
I20260925 15:07:16.464680 4408 raft_consensus.cc:2258] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:16.465349 4407 tablet_replica.cc:330] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185: stopping tablet replica
I20260925 15:07:16.465574 4406 ts_tablet_manager.cc:1905] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:16.465713 4407 raft_consensus.cc:2229] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:16.466140 4407 raft_consensus.cc:2258] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:16.467886 4408 ts_tablet_manager.cc:1905] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:16.470073 4407 ts_tablet_manager.cc:1905] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:16.472564 4406 ts_tablet_manager.cc:1918] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:16.472749 4406 log.cc:1196] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/9372df41ed0b46bebc4dac825dcd0cd9
I20260925 15:07:16.473289 4406 ts_tablet_manager.cc:1939] T 9372df41ed0b46bebc4dac825dcd0cd9 P 2c0f8dbc4f7c46d889ee23657d132d4d: Deleting consensus metadata
I20260925 15:07:16.475245 3860 catalog_manager.cc:4841] TS 2c0f8dbc4f7c46d889ee23657d132d4d (127.22.236.1:41107): tablet 9372df41ed0b46bebc4dac825dcd0cd9 (table test-table [id=46f74c68b30641fd9a99cee1b85bbdc9]) successfully deleted
I20260925 15:07:16.478679 4408 ts_tablet_manager.cc:1918] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:16.478821 4408 log.cc:1196] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/9372df41ed0b46bebc4dac825dcd0cd9
I20260925 15:07:16.479295 4408 ts_tablet_manager.cc:1939] T 9372df41ed0b46bebc4dac825dcd0cd9 P c71f5f4472864248a9034703d33e1b3e: Deleting consensus metadata
I20260925 15:07:16.480916 3861 catalog_manager.cc:4841] TS c71f5f4472864248a9034703d33e1b3e (127.22.236.2:44903): tablet 9372df41ed0b46bebc4dac825dcd0cd9 (table test-table [id=46f74c68b30641fd9a99cee1b85bbdc9]) successfully deleted
I20260925 15:07:16.482062 4407 ts_tablet_manager.cc:1918] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:16.482281 4407 log.cc:1196] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/9372df41ed0b46bebc4dac825dcd0cd9
I20260925 15:07:16.482801 4407 ts_tablet_manager.cc:1939] T 9372df41ed0b46bebc4dac825dcd0cd9 P b51ab5b963eb48f296fea4ff4be4d185: Deleting consensus metadata
I20260925 15:07:16.484458 3863 catalog_manager.cc:4841] TS b51ab5b963eb48f296fea4ff4be4d185 (127.22.236.3:39709): tablet 9372df41ed0b46bebc4dac825dcd0cd9 (table test-table [id=46f74c68b30641fd9a99cee1b85bbdc9]) successfully deleted
I20260925 15:07:16.494567 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:16.466953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.22.236.1:41107)
0925 15:07:16.467129 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:16.467134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:16.479369 (+ 12235us) server_negotiation.cc:407] Connection header received
0925 15:07:16.481019 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:16.481025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:16.481104 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:16.481173 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:16.483162 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.484265 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.485580 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.485982 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.486084 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:16.488534 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:16.488551 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:16.488569 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:16.488616 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:16.491700 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:16.492586 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:16.492595 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:16.492599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:16.492730 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:16.493086 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:16.493090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:16.493093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:16.493429 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:16.493720 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:16.493994 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 15:07:16.494210 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:07:16.495157 4068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:16.583705 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 3913
I20260925 15:07:16.603296 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 4077
I20260925 15:07:16.618326 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:16.604092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.22.236.2:44903)
0925 15:07:16.604316 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:16.604322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:16.604343 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:07:16.604410 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:16.604416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:16.604516 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:16.604623 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:16.605879 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.606977 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.608643 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.609316 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.609440 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:16.611776 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:16.611794 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:16.611815 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:16.611870 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:16.615570 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:16.616254 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:16.616264 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:16.616268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:16.616422 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:16.616747 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:16.616752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:16.616756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:16.617190 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:16.617495 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:16.617712 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 15:07:16.617927 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 15:07:16.619093 4212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:16.729707 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 4077
I20260925 15:07:16.743398 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 4222
I20260925 15:07:16.756024 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:16.744096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.22.236.3:39709)
0925 15:07:16.744290 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:16.744296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:16.744316 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:07:16.744477 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:16.744483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:16.744552 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:16.744632 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:16.746225 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.746912 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.748110 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.748534 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.748640 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:16.751031 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:16.751045 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:16.751061 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:16.751107 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:16.753427 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:16.754026 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:16.754033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:16.754035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:16.754158 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:16.754501 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:16.754510 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:16.754512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:16.754896 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:16.755160 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:16.755463 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 15:07:16.755671 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:07:16.756696 4346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:16.856124 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 4222
I20260925 15:07:16.869079 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 3843
I20260925 15:07:16.870167 3906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:16.978646 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 3843
I20260925 15:07:16.995736 23472 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:07:17.330084 23472 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:07:17.330291 23472 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:07:17.664490 23472 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:07:17.664 UTC [2895] LOG: received smart shutdown request
2026-09-25 15:07:17.667 UTC [2895] LOG: background worker "logical replication launcher" (PID 2904) exited with exit code 1
2026-09-25 15:07:17.667 UTC [2899] LOG: shutting down
2026-09-25 15:07:17.822 UTC [2895] LOG: database system is shut down
2026-09-25T15:07:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51090 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:18 dist-test-slave-nsrc krb5kdc[4422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:18 dist-test-slave-nsrc krb5kdc[4422](info): set up 2 sockets
Sep 25 15:07:18 dist-test-slave-nsrc krb5kdc[4422](info): commencing operation
krb5kdc: starting...
W20260925 15:07:18.920361 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:07:19 dist-test-slave-nsrc krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:07:19.121918 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:39779
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:39779,127.22.236.61:38475,127.22.236.60:46543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:19.434185 4435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:19.434464 4435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:19.434594 4435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:19.441670 4435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:19.441771 4435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:19.441844 4435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:19.441918 4435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:19.441973 4435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:19.451495 4435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:39779,127.22.236.61:38475,127.22.236.60:46543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:39779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:19.452785 4435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:19.454044 4435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:19.465113 4440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:19.465042 4441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:19.465085 4443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:19.466128 4435 server_base.cc:1034] running on GCE node
I20260925 15:07:19.466838 4435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:19.468118 4435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:19.469357 4435 hybrid_clock.cc:648] HybridClock initialized: now 1790348839469314 us; error 28 us; skew 500 ppm
I20260925 15:07:19.469676 4435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:19 dist-test-slave-nsrc krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:19.472545 4435 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:07:19.473923 4435 webserver.cc:460] Webserver started at http://127.22.236.62:46243/ using document root <none> and password file <none>
I20260925 15:07:19.474546 4435 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:19.474658 4435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:19.474925 4435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:19.477753 4435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "a168ef653eff463bb88f94f6d019893f"
format_stamp: "Formatted at 2026-09-25 15:07:19 on dist-test-slave-nsrc"
server_key: "96ec0ef06de72756800a7d97e0593f03"
server_key_iv: "1ab90ef679e42a32b89b705525939f96"
server_key_version: "encryptionkey@0"
I20260925 15:07:19.478513 4435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "a168ef653eff463bb88f94f6d019893f"
format_stamp: "Formatted at 2026-09-25 15:07:19 on dist-test-slave-nsrc"
server_key: "96ec0ef06de72756800a7d97e0593f03"
server_key_iv: "1ab90ef679e42a32b89b705525939f96"
server_key_version: "encryptionkey@0"
I20260925 15:07:19.483278 4435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 15:07:19.486572 4450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:19.487828 4435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:07:19.487978 4435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "a168ef653eff463bb88f94f6d019893f"
format_stamp: "Formatted at 2026-09-25 15:07:19 on dist-test-slave-nsrc"
server_key: "96ec0ef06de72756800a7d97e0593f03"
server_key_iv: "1ab90ef679e42a32b89b705525939f96"
server_key_version: "encryptionkey@0"
I20260925 15:07:19.488183 4435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:19.509414 4435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:19.513167 4435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:19.513420 4435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:19.529867 4435 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:39779
I20260925 15:07:19.529901 4501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:39779 every 8 connection(s)
I20260925 15:07:19.531116 4435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:19.534492 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:19.538820 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 4435
I20260925 15:07:19.539045 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:07:19.539814 23472 external_mini_cluster.cc:1428] Setting key bcc624da47cd0d7caa2057bdca731529
I20260925 15:07:19.541846 4502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:07:19 dist-test-slave-nsrc krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.236.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.61@KRBTEST.COM" created.
I20260925 15:07:19.565331 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:19.545548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.22.236.62:39779)
0925 15:07:19.547351 (+ 1803us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:19.547359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:19.547400 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:07:19.549116 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:19.549124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:19.549412 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:19.549612 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:19.550696 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:19.552487 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:19.553839 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:19.554641 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:19.554797 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:19.558414 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:19.558427 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:19.558445 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:19.558504 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:19.561252 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:19.561865 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:19.561871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:19.561873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:19.562013 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:19.562440 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:19.562443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:19.562445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:19.563083 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:19.563263 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:19.563899 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 15:07:19.564306 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":201,"server-negotiator.queue_time_us":1611,"spinlock_wait_cycles":77056,"thread_start_us":1388,"threads_started":1}
I20260925 15:07:19.565363 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:19.545548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:39779 (local address 127.0.0.1:58222)
0925 15:07:19.546426 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:19.546480 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:19.548719 (+ 2239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:19.549638 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:19.549643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:19.550171 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:19.550498 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:19.550516 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:19.552627 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:19.552630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:19.553698 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:19.553714 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:19.554847 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:19.554851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:19.555033 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:19.555634 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:19.555660 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:19.558093 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:19.561372 (+ 3279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:19.561382 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:19.561391 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:19.561760 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:19.562123 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:19.562127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:19.562131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:19.562325 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:19.563289 (+ 964us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:19.563296 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:19.563514 (+ 218us) client_negotiation.cc:769] Sending connection context
0925 15:07:19.563739 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 15:07:19.564183 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":41,"thread_start_us":368,"threads_started":1}
I20260925 15:07:19.568344 4502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:07:19.569442 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:19.569016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:38475 (local address 127.0.0.1:45758)
0925 15:07:19.569191 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:19.569314 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 15:07:19.569736 4451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
I20260925 15:07:19.571183 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:19.570826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:38475 (local address 127.0.0.1:45770)
0925 15:07:19.570968 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:19.571045 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 15:07:19.571389 4502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:38475: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.61.keytab.
Entry for principal kudu/127.22.236.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.61.keytab.
I20260925 15:07:19.587110 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.61:38475
--webserver_interface=127.22.236.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.61.keytab
--principal=kudu/127.22.236.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:39779,127.22.236.61:38475,127.22.236.60:46543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:07:19.620661 4502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } attempt: 1
I20260925 15:07:19.621768 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:19.621344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:38475 (local address 127.0.0.1:45772)
0925 15:07:19.621528 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:19.621638 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 15:07:19.622723 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:19.622431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:38475 (local address 127.0.0.1:45784)
0925 15:07:19.622541 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:19.622603 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 15:07:19.622896 4502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:38475: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
I20260925 15:07:19.691121 4502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } attempt: 2
I20260925 15:07:19.692305 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:19.691809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:38475 (local address 127.0.0.1:45800)
0925 15:07:19.692029 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:19.692131 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 15:07:19.693274 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:19.692950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:38475 (local address 127.0.0.1:45808)
0925 15:07:19.693070 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:19.693150 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:07:19.693456 4502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:38475: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
I20260925 15:07:19.784686 4502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } attempt: 3
I20260925 15:07:19.785806 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:19.785387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:38475 (local address 127.0.0.1:45814)
0925 15:07:19.785578 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:19.785681 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 15:07:19.786741 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:19.786435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:38475 (local address 127.0.0.1:45822)
0925 15:07:19.786546 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:19.786611 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 15:07:19.786924 4502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:38475: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
W20260925 15:07:19.896229 4509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:19.896499 4509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:19.896627 4509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:19.903644 4509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:19.903745 4509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:19.903816 4509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:19.903895 4509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:19.903954 4509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:19.913617 4509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:39779,127.22.236.61:38475,127.22.236.60:46543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.61.keytab
--principal=kudu/127.22.236.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.61:38475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.236.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:19.914912 4509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:19.916203 4509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:19.927062 4514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:19.927116 4515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:19.927160 4517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:19.928287 4509 server_base.cc:1034] running on GCE node
I20260925 15:07:19.929030 4509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:19.930119 4502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } attempt: 4
I20260925 15:07:19.930591 4509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:19.931197 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:19.930659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:38475 (local address 127.0.0.1:45824)
0925 15:07:19.930984 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:19.931065 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260925 15:07:19.931780 4509 hybrid_clock.cc:648] HybridClock initialized: now 1790348839931722 us; error 41 us; skew 500 ppm
I20260925 15:07:19.932114 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:19.931744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:38475 (local address 127.0.0.1:45834)
0925 15:07:19.931892 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:19.931997 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:07:19.932130 4509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20260925 15:07:19.932314 4502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:38475: Network error: Client connection negotiation failed: client connection to 127.22.236.61:38475: connect: Connection refused (error 111)
Sep 25 15:07:19 dist-test-slave-nsrc krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:19.939319 4509 init.cc:377] Logged in from keytab as kudu/127.22.236.61@KRBTEST.COM (short username kudu)
I20260925 15:07:19.940667 4509 webserver.cc:460] Webserver started at http://127.22.236.61:45315/ using document root <none> and password file <none>
I20260925 15:07:19.941305 4509 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:19.941401 4509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:19.941638 4509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:19.944490 4509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/data/instance:
uuid: "7b26119fa68e4478afb9f3cd5188fa8a"
format_stamp: "Formatted at 2026-09-25 15:07:19 on dist-test-slave-nsrc"
server_key: "f17eb6d8c4d01b4dcc82abee512ecdf8"
server_key_iv: "ad2f0b8390ca208dbe0afec511ec0e21"
server_key_version: "encryptionkey@0"
I20260925 15:07:19.945245 4509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/wal/instance:
uuid: "7b26119fa68e4478afb9f3cd5188fa8a"
format_stamp: "Formatted at 2026-09-25 15:07:19 on dist-test-slave-nsrc"
server_key: "f17eb6d8c4d01b4dcc82abee512ecdf8"
server_key_iv: "ad2f0b8390ca208dbe0afec511ec0e21"
server_key_version: "encryptionkey@0"
I20260925 15:07:19.949996 4509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 15:07:19.953166 4524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:19.954480 4509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:07:19.954613 4509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/wal
uuid: "7b26119fa68e4478afb9f3cd5188fa8a"
format_stamp: "Formatted at 2026-09-25 15:07:19 on dist-test-slave-nsrc"
server_key: "f17eb6d8c4d01b4dcc82abee512ecdf8"
server_key_iv: "ad2f0b8390ca208dbe0afec511ec0e21"
server_key_version: "encryptionkey@0"
I20260925 15:07:19.954777 4509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:19.976087 4509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:19.980089 4509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:19.980386 4509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:19.996831 4509 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.61:38475
I20260925 15:07:19.996860 4575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.61:38475 every 8 connection(s)
I20260925 15:07:19.998116 4509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/data/info.pb
I20260925 15:07:20.001685 4576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:20.004400 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 4509
I20260925 15:07:20.004577 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-1/wal/instance
I20260925 15:07:20.005002 23472 external_mini_cluster.cc:1428] Setting key db549cf2eefa3167e6a881c47b04e7d2
I20260925 15:07:20.008104 4576 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:07:20 dist-test-slave-nsrc krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.61@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.236.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.60@KRBTEST.COM" created.
I20260925 15:07:20.027956 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.010705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58226 (local address 127.22.236.62:39779)
0925 15:07:20.010915 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.010921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.012091 (+ 1170us) server_negotiation.cc:407] Connection header received
0925 15:07:20.013896 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.013902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.013982 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.014088 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.015474 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.016486 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.017920 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.018554 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.018680 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.022072 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.022079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.022084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.022124 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.025010 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.025822 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.025827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.025829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.025956 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.026363 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.026365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.026367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.026640 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.026747 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.027553 (+ 806us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.027724 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:07:20.029256 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.011171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:39779 (local address 127.0.0.1:58226)
0925 15:07:20.011922 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.011972 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:20.013648 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:20.014272 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:20.014279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:20.014775 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:20.015331 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.015349 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.016620 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.016623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.017791 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.017802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.018749 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.018754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.018932 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.019601 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:20.019621 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:20.021833 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:20.025140 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.025149 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.025165 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.025702 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.026045 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.026057 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.026061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.026278 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.026748 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:20.026755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:20.027154 (+ 399us) client_negotiation.cc:769] Sending connection context
0925 15:07:20.027438 (+ 284us) client_negotiation.cc:240] Negotiation successful
0925 15:07:20.027895 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":313,"threads_started":1}
I20260925 15:07:20.030856 4576 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:07:20 dist-test-slave-nsrc krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.61@KRBTEST.COM for kudu/127.22.236.61@KRBTEST.COM
I20260925 15:07:20.045287 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.031597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:38475 (local address 127.0.0.1:45836)
0925 15:07:20.031799 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.031818 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:20.031932 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:20.032826 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:20.032829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:20.033159 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:20.033296 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.033304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.035038 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.035042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.035873 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.035884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.036984 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.036988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.037115 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.037624 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:20.037641 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:20.039709 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:20.042794 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.042799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.042803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.043260 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.043623 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.043627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.043629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.043760 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.044463 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:20.044466 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:20.044584 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 15:07:20.044689 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 15:07:20.044940 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 15:07:20.045419 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.031767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.22.236.61:38475)
0925 15:07:20.032403 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.032408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.032433 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:07:20.032471 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.032476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.032684 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.032810 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.033438 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.034908 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.036006 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.036810 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.036914 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.039996 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.040008 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.040021 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.040054 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.042662 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.043382 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.043389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.043392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.043515 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.043873 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.043877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.043879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.044331 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.044506 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.044827 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.045063 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":365,"threads_started":1}
I20260925 15:07:20.047672 4576 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:07:20.048686 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.048279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33500)
0925 15:07:20.048432 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.048538 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:07:20.048894 4525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
I20260925 15:07:20.050246 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.049963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33506)
0925 15:07:20.050080 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.050154 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:07:20.050442 4576 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:46543: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.60.keytab.
Entry for principal kudu/127.22.236.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.60.keytab.
I20260925 15:07:20.051542 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.60:46543
--webserver_interface=127.22.236.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.60.keytab
--principal=kudu/127.22.236.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:39779,127.22.236.61:38475,127.22.236.60:46543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:07:20.099769 4576 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } attempt: 1
I20260925 15:07:20.101027 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.100525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33510)
0925 15:07:20.100738 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.100872 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 15:07:20.102018 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.101675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33522)
0925 15:07:20.101799 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.101912 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 15:07:20.102209 4576 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:46543: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
I20260925 15:07:20.170430 4576 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } attempt: 2
I20260925 15:07:20.171492 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.171056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33526)
0925 15:07:20.171251 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.171337 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 15:07:20.172535 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.172228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33532)
0925 15:07:20.172346 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.172421 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 15:07:20.172732 4576 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:46543: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
I20260925 15:07:20.231563 4502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } attempt: 5
Sep 25 15:07:20 dist-test-slave-nsrc krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for kudu/127.22.236.61@KRBTEST.COM
I20260925 15:07:20.244107 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.232342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.22.236.61:38475)
0925 15:07:20.232518 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.232524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.232656 (+ 132us) server_negotiation.cc:407] Connection header received
0925 15:07:20.232840 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.232844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.232956 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.233045 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.233833 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.234533 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.235674 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.236128 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.236279 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.239208 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.239213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.239216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.239248 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.241699 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.242316 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.242321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.242323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.242441 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.242782 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.242787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.242791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.243081 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.243211 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.243587 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.243796 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:07:20.244104 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.232343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:38475 (local address 127.0.0.1:45846)
0925 15:07:20.232566 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.232583 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:20.232720 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:20.233109 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:20.233114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:20.233500 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:20.233673 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.233682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.234664 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.234669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.235535 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.235547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.236294 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.236300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.236493 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.237086 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:20.237105 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:20.239024 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:20.241840 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.241846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.241850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.242206 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.242544 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.242549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.242553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.242685 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.243192 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:20.243197 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:20.243348 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 15:07:20.243475 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 15:07:20.243750 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260925 15:07:20.245316 4502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:07:20.246387 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.245992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33538)
0925 15:07:20.246165 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.246243 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 15:07:20.247354 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.247022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33550)
0925 15:07:20.247164 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.247253 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:07:20.247646 4502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:46543: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
I20260925 15:07:20.263931 4576 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } attempt: 3
I20260925 15:07:20.265026 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.264574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33558)
0925 15:07:20.264766 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.264868 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 15:07:20.266119 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.265683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33560)
0925 15:07:20.265912 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.265991 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260925 15:07:20.266297 4576 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:46543: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
I20260925 15:07:20.298933 4502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } attempt: 1
I20260925 15:07:20.300083 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.299638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33568)
0925 15:07:20.299821 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.299923 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 15:07:20.301077 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.300743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33580)
0925 15:07:20.300893 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.300967 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:07:20.301275 4502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:46543: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
W20260925 15:07:20.359476 4583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:20.359771 4583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:20.359905 4583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:20.366953 4583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:20.367066 4583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:20.367146 4583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:20.367218 4583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:20.367278 4583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:20.369473 4502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } attempt: 2
I20260925 15:07:20.370509 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.370056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33590)
0925 15:07:20.370255 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.370347 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 15:07:20.371516 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.371163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33602)
0925 15:07:20.371313 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.371382 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:07:20.371694 4502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:46543: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
I20260925 15:07:20.377099 4583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:39779,127.22.236.61:38475,127.22.236.60:46543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.60.keytab
--principal=kudu/127.22.236.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.60:46543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.236.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:20.378357 4583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:20.379629 4583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:20.390617 4588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:20.390566 4589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:20.391302 4583 server_base.cc:1034] running on GCE node
W20260925 15:07:20.391418 4591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:20.392282 4583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:20.393703 4583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:20.394891 4583 hybrid_clock.cc:648] HybridClock initialized: now 1790348840394851 us; error 41 us; skew 500 ppm
I20260925 15:07:20.395218 4583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:20 dist-test-slave-nsrc krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348840, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:20.398222 4583 init.cc:377] Logged in from keytab as kudu/127.22.236.60@KRBTEST.COM (short username kudu)
I20260925 15:07:20.399695 4583 webserver.cc:460] Webserver started at http://127.22.236.60:38075/ using document root <none> and password file <none>
I20260925 15:07:20.400430 4583 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:20.400542 4583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:20.400823 4583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:20.403689 4583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/data/instance:
uuid: "8bf02686bc2d42c38cb2f943be5832d4"
format_stamp: "Formatted at 2026-09-25 15:07:20 on dist-test-slave-nsrc"
server_key: "76d248ce3755d6c8ad3b238c29dfbcda"
server_key_iv: "892ab8fa59d5de01223a0849fb2c4199"
server_key_version: "encryptionkey@0"
I20260925 15:07:20.404505 4583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/wal/instance:
uuid: "8bf02686bc2d42c38cb2f943be5832d4"
format_stamp: "Formatted at 2026-09-25 15:07:20 on dist-test-slave-nsrc"
server_key: "76d248ce3755d6c8ad3b238c29dfbcda"
server_key_iv: "892ab8fa59d5de01223a0849fb2c4199"
server_key_version: "encryptionkey@0"
I20260925 15:07:20.409469 4583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 15:07:20.409523 4576 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } attempt: 4
I20260925 15:07:20.410526 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.410073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33610)
0925 15:07:20.410290 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.410371 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 15:07:20.411450 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.411119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33620)
0925 15:07:20.411264 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.411329 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 15:07:20.411645 4576 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:46543: Network error: Client connection negotiation failed: client connection to 127.22.236.60:46543: connect: Connection refused (error 111)
I20260925 15:07:20.412833 4598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:20.414224 4583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:07:20.414371 4583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/wal
uuid: "8bf02686bc2d42c38cb2f943be5832d4"
format_stamp: "Formatted at 2026-09-25 15:07:20 on dist-test-slave-nsrc"
server_key: "76d248ce3755d6c8ad3b238c29dfbcda"
server_key_iv: "892ab8fa59d5de01223a0849fb2c4199"
server_key_version: "encryptionkey@0"
I20260925 15:07:20.414533 4583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:20.428980 4583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:20.432854 4583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:20.433112 4583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:20.449615 4583 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.60:46543
I20260925 15:07:20.449637 4649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.60:46543 every 8 connection(s)
I20260925 15:07:20.450930 4583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/data/info.pb
I20260925 15:07:20.454459 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:20.460312 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 4583
I20260925 15:07:20.460467 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348645297121-23472-0/minicluster-data/master-2/wal/instance
I20260925 15:07:20.460872 23472 external_mini_cluster.cc:1428] Setting key 5cf862e41d7ffce2871109a603f596f0
I20260925 15:07:20.460904 4650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:07:20 dist-test-slave-nsrc krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348840, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.60@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
Sep 25 15:07:20 dist-test-slave-nsrc krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.60@KRBTEST.COM
I20260925 15:07:20.477875 4502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } attempt: 3
I20260925 15:07:20.479388 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.463484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.22.236.62:39779)
0925 15:07:20.463711 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.463718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.464400 (+ 682us) server_negotiation.cc:407] Connection header received
0925 15:07:20.465695 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.465701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.465775 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.465857 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.467040 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.467731 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.469178 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.469602 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.469709 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.473527 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.473532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.473536 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.473568 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.476233 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.476965 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.476969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.476971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.477114 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.477516 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.477520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.477522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.477826 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.477980 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.478775 (+ 795us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.478988 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:07:20.480628 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.463486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:39779 (local address 127.0.0.1:58228)
0925 15:07:20.464276 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.464312 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:20.465470 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:20.465931 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:20.465937 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:20.466526 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:20.466886 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.466902 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.467870 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.467874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.469043 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.469064 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.469800 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.469807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.470058 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.470766 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:20.470796 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:20.473349 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:20.476349 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.476364 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.476377 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.476848 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.477213 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.477218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.477222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.477406 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.477967 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:20.477974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:20.478364 (+ 390us) client_negotiation.cc:769] Sending connection context
0925 15:07:20.478675 (+ 311us) client_negotiation.cc:240] Negotiation successful
0925 15:07:20.479203 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":34,"thread_start_us":412,"threads_started":1}
I20260925 15:07:20.481475 4653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.462615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.22.236.60:46543)
0925 15:07:20.463336 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.463349 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.463398 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:07:20.465465 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.465485 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.465706 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.465879 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.467313 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.468480 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.469801 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.470431 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.470561 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.474836 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.474844 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.474855 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.474884 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.477569 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.478336 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.478343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.478346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.478865 (+ 519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.479323 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.479328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.479331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.479762 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.480061 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.480705 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.480960 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":528,"thread_start_us":321,"threads_started":1}
I20260925 15:07:20.482800 4650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:07:20 dist-test-slave-nsrc krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for kudu/127.22.236.60@KRBTEST.COM
Sep 25 15:07:20 dist-test-slave-nsrc krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348840, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.60@KRBTEST.COM for kudu/127.22.236.61@KRBTEST.COM
Sep 25 15:07:20 dist-test-slave-nsrc krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:20.497453 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.483558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:38475 (local address 127.0.0.1:45848)
0925 15:07:20.483753 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.483769 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:20.483861 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:20.484248 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:20.484252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:20.484719 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:20.484937 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.484947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.485850 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.485855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.486847 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.486861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.488042 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.488047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.488251 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.488708 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:20.488724 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:20.491181 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:20.494680 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.494685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.494688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.495143 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.495763 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.495768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.495771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.495910 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.496569 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:20.496573 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:20.496690 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 15:07:20.496802 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 15:07:20.497072 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":50}
I20260925 15:07:20.497473 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.483584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45848 (local address 127.22.236.61:38475)
0925 15:07:20.483737 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.483742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.483964 (+ 222us) server_negotiation.cc:407] Connection header received
0925 15:07:20.484031 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.484037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.484109 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.484216 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.485084 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.485728 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.487007 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.487910 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.488024 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.491378 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.491385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.491390 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.491432 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.494519 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.495264 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.495271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.495274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.495457 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.496028 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.496032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.496034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.496372 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.496512 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.496910 (+ 398us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.497109 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:07:20.497531 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.478403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33642)
0925 15:07:20.478581 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.478601 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:20.478691 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:20.482119 (+ 3428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:20.482124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:20.482698 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:20.482895 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.482907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.484732 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.484738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.486183 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.486199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.487153 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.487159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.487344 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.487982 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:20.488025 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:20.490792 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:20.494816 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.494821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.494825 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.495359 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.495694 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.495699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.495704 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.495870 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.496550 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:20.496556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:20.496724 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 15:07:20.496865 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 15:07:20.497157 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":1}
I20260925 15:07:20.498158 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.478813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.22.236.60:46543)
0925 15:07:20.481845 (+ 3032us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.481849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.481870 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:07:20.481922 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.481926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.481993 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.482091 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.483082 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.484589 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.486372 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.486997 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.487661 (+ 664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.490985 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.490990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.490993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.491022 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.494688 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.495476 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.495479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.495481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.495583 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.495991 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.495996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.496000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.496380 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.497181 (+ 801us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.497302 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.497512 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":2905,"thread_start_us":2746,"threads_started":1}
I20260925 15:07:20.498764 4650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:07:20.501468 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.484493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.22.236.62:39779)
0925 15:07:20.484639 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.484644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.484709 (+ 65us) server_negotiation.cc:407] Connection header received
0925 15:07:20.484914 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.484919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.484986 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.485070 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.486651 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.487460 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.489152 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.489772 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.489885 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.494162 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.494166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.494169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.494198 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.497962 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.499181 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.499187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.499191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.499338 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.499761 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.499766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.499769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.500105 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.500390 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.500918 (+ 528us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.501139 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:07:20.503825 4502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f: Bootstrap starting.
Sep 25 15:07:20 dist-test-slave-nsrc krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348840, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.60@KRBTEST.COM for kudu/127.22.236.60@KRBTEST.COM
I20260925 15:07:20.507927 4502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:20.509258 4502 log.cc:824] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:07:20 dist-test-slave-nsrc krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.61@KRBTEST.COM
I20260925 15:07:20.511947 4502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f: No bootstrap required, opened a new log
I20260925 15:07:20.513270 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.499602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33656)
0925 15:07:20.499758 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.499775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:20.499859 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:20.500252 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:20.500257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:20.500673 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:20.500847 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.500859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.501901 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.501906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.503083 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.503097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.504096 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.504101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.504286 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.504980 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:20.505001 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:20.507551 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:20.510788 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.510792 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.510796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.511266 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.511646 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.511650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.511654 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.511787 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.512393 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:20.512397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:20.512530 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 15:07:20.512651 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 15:07:20.512928 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:07:20.513661 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.499802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.22.236.60:46543)
0925 15:07:20.499953 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.499957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.499974 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:07:20.500017 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.500022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.500084 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.500226 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.500980 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.501793 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.503222 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.503815 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.503933 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.507738 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.507744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.507748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.507779 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.510642 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.511389 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.511394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.511397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.511533 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.511905 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.511909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.511911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.512283 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.512404 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.512759 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.513207 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":252,"server-negotiator.queue_time_us":48}
I20260925 15:07:20.518429 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.503445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.22.236.61:38475)
0925 15:07:20.503608 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.503614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.503630 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:07:20.503676 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.503681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.503738 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.503810 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.505109 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.505798 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.506962 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.507438 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.507541 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.512099 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.512105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.512109 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.512175 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.515507 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.516316 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.516321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.516326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.516485 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.516939 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.516944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.516947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.517283 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.517546 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.517884 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.518107 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:07:20.518505 4650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4: Bootstrap starting.
I20260925 15:07:20.521095 4502 raft_consensus.cc:348] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } }
I20260925 15:07:20.521463 4502 raft_consensus.cc:374] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:20.521588 4502 raft_consensus.cc:729] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a168ef653eff463bb88f94f6d019893f, State: Initialized, Role: FOLLOWER
I20260925 15:07:20.522167 4650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:20.522217 4502 consensus_queue.cc:260] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } }
I20260925 15:07:20.523327 4650 log.cc:824] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:20.523576 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:07:20.525892 4650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4: No bootstrap required, opened a new log
I20260925 15:07:20.528692 4657 sys_catalog.cc:455] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } } }
I20260925 15:07:20.529196 4657 sys_catalog.cc:458] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:07:20.533619 4502 sys_catalog.cc:564] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:20.533839 4650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } }
I20260925 15:07:20.534425 4650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:20.534680 4650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf02686bc2d42c38cb2f943be5832d4, State: Initialized, Role: FOLLOWER
I20260925 15:07:20.535542 4650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } }
I20260925 15:07:20.548171 4664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } } }
I20260925 15:07:20.548713 4664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:07:20.549687 4650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:20.570868 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.549637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.22.236.61:38475)
0925 15:07:20.549802 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.549807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.552552 (+ 2745us) server_negotiation.cc:407] Connection header received
0925 15:07:20.552864 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.552872 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.552963 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.553035 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.555494 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.556694 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.559100 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.559774 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.559881 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.563976 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.563984 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.563990 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.564037 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.567565 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.568477 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.568484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.568488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.568656 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.569127 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.569132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.569136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.569525 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.569743 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.570288 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.570515 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:07:20.584599 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.549278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.22.236.62:39779)
0925 15:07:20.549452 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.549458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.560580 (+ 11122us) server_negotiation.cc:407] Connection header received
0925 15:07:20.561565 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.561571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.561660 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.561730 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.565323 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.566450 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.570446 (+ 3996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.571110 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.571201 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.577314 (+ 6113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.577321 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.577326 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.577365 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.581041 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.581945 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.581950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.581953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.582357 (+ 404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.582800 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.582805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.582808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.583145 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.583378 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.583954 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.584213 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:07:20.588121 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.561241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.22.236.60:46543)
0925 15:07:20.561426 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.561433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.570060 (+ 8627us) server_negotiation.cc:407] Connection header received
0925 15:07:20.572158 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.572165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.572269 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.572344 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.574849 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.575961 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.577853 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.578527 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.578622 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.581515 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.581521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.581525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.581565 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.584960 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.585839 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.585844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.585847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.585997 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.586424 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.586429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.586431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.586761 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.587027 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.587577 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.587786 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 15:07:20.593664 4686 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:07:20.593837 4686 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:07:20.594882 4435 master_runner.cc:186] Error getting master registration for 127.22.236.61:38475: OK, instance_id { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" instance_seqno: 1790348839992230 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:07:20.607980 4435 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:07:20.608356 4657 raft_consensus.cc:482] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:20.608601 4657 raft_consensus.cc:504] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } }
I20260925 15:07:20.609994 4657 leader_election.cc:288] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b26119fa68e4478afb9f3cd5188fa8a (127.22.236.61:38475), 8bf02686bc2d42c38cb2f943be5832d4 (127.22.236.60:46543)
W20260925 15:07:20.610797 4689 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:07:20.611008 4689 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:07:20.610742 4625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a168ef653eff463bb88f94f6d019893f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf02686bc2d42c38cb2f943be5832d4" is_pre_election: true
I20260925 15:07:20.611377 4625 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a168ef653eff463bb88f94f6d019893f in term 0.
I20260925 15:07:20.612730 4451 leader_election.cc:302] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bf02686bc2d42c38cb2f943be5832d4, a168ef653eff463bb88f94f6d019893f; no voters:
I20260925 15:07:20.613236 4657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:20.613374 4657 raft_consensus.cc:482] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:20.613473 4657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:20.602416 4583 master_runner.cc:186] Error getting master registration for 127.22.236.61:38475: OK, instance_id { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" instance_seqno: 1790348839992230 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:07:20.616748 4657 raft_consensus.cc:504] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } }
I20260925 15:07:20.617105 4583 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:07:20.617405 4657 leader_election.cc:288] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [CANDIDATE]: Term 1 election: Requested vote from peers 7b26119fa68e4478afb9f3cd5188fa8a (127.22.236.61:38475), 8bf02686bc2d42c38cb2f943be5832d4 (127.22.236.60:46543)
I20260925 15:07:20.618028 4625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a168ef653eff463bb88f94f6d019893f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf02686bc2d42c38cb2f943be5832d4"
I20260925 15:07:20.618243 4625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:20.621644 4625 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a168ef653eff463bb88f94f6d019893f in term 1.
I20260925 15:07:20.623363 4451 leader_election.cc:302] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bf02686bc2d42c38cb2f943be5832d4, a168ef653eff463bb88f94f6d019893f; no voters:
I20260925 15:07:20.623734 4657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:20.624415 4657 raft_consensus.cc:686] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [term 1 LEADER]: Becoming Leader. State: Replica: a168ef653eff463bb88f94f6d019893f, State: Running, Role: LEADER
I20260925 15:07:20.625026 4657 consensus_queue.cc:237] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } }
I20260925 15:07:20.642709 4691 sys_catalog.cc:455] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a168ef653eff463bb88f94f6d019893f. Latest consensus state: current_term: 1 leader_uuid: "a168ef653eff463bb88f94f6d019893f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } } }
I20260925 15:07:20.643445 4691 sys_catalog.cc:458] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:20.644650 4693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:20.646845 4693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:20.657963 4625 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [term 1 FOLLOWER]: Refusing update from remote peer a168ef653eff463bb88f94f6d019893f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:20.659194 4691 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:20.677506 4664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a168ef653eff463bb88f94f6d019893f. Latest consensus state: current_term: 1 leader_uuid: "a168ef653eff463bb88f94f6d019893f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } } }
I20260925 15:07:20.677835 4664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:07:20.680999 4694 mvcc.cc:205] Tried to move back new op lower bound from 7333268851322810368 to 7333268851205505024. Current Snapshot: MvccSnapshot[applied={T|T < 7333268851322810368}]
I20260925 15:07:20.682761 4664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a168ef653eff463bb88f94f6d019893f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } } }
I20260925 15:07:20.683000 4664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:07:20.684996 4604 tablet.cc:2369] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:07:20.686839 4695 mvcc.cc:205] Tried to move back new op lower bound from 7333268851322810368 to 7333268851205505024. Current Snapshot: MvccSnapshot[applied={T|T < 7333268851322810368}]
I20260925 15:07:20.693063 4691 sys_catalog.cc:455] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a168ef653eff463bb88f94f6d019893f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } } }
I20260925 15:07:20.693369 4691 sys_catalog.cc:458] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:20.694551 4693 catalog_manager.cc:1347] Generated new cluster ID: 1fd1c89b06894b4094f8651c3701e66c
I20260925 15:07:20.694651 4693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:20.710855 4576 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } attempt: 5
I20260925 15:07:20.718097 4693 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:20.720073 4693 catalog_manager.cc:1504] Loading token signing keys...
Sep 25 15:07:20 dist-test-slave-nsrc krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.61@KRBTEST.COM for kudu/127.22.236.60@KRBTEST.COM
I20260925 15:07:20.734578 4693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f: Generated new TSK 0
I20260925 15:07:20.735620 4693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:07:20.742853 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.712068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.22.236.60:46543)
0925 15:07:20.712271 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.712277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.712293 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:07:20.712346 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.712352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.712427 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.712534 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.713476 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.714633 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.716426 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.717182 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.717306 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.725858 (+ 8552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.725865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.725871 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.725909 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.729749 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.732772 (+ 3023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.732834 (+ 62us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.732839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.733714 (+ 875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.738310 (+ 4596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.738318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.738322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.739234 (+ 912us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.739510 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.740652 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.741188 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 15:07:20.744206 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.711394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:46543 (local address 127.0.0.1:33676)
0925 15:07:20.711737 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.711757 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:20.711877 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:20.712582 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:20.712588 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:20.713108 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:20.713308 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.713320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.714759 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.714765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.716220 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.716235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.717549 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.717556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.717777 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.718666 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:20.718690 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:20.725460 (+ 6770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:20.731468 (+ 6008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.731473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.731512 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.732542 (+ 1030us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.737685 (+ 5143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.737691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.737695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.738036 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.739668 (+ 1632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:20.739673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:20.740000 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 15:07:20.741262 (+ 1262us) client_negotiation.cc:240] Negotiation successful
0925 15:07:20.742634 (+ 1372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260925 15:07:20.750190 4576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a: Bootstrap starting.
I20260925 15:07:20.764009 4576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:20.767117 4576 log.cc:824] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:20.783185 4576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a: No bootstrap required, opened a new log
I20260925 15:07:20.808025 4576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } }
I20260925 15:07:20.809368 4576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:20.809718 4576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b26119fa68e4478afb9f3cd5188fa8a, State: Initialized, Role: FOLLOWER
I20260925 15:07:20.811317 4576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } }
I20260925 15:07:20.824327 4576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:20.825480 4550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a168ef653eff463bb88f94f6d019893f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26119fa68e4478afb9f3cd5188fa8a"
I20260925 15:07:20.826130 4551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a168ef653eff463bb88f94f6d019893f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" is_pre_election: true
I20260925 15:07:20.826053 4705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } } }
I20260925 15:07:20.826828 4705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:07:20.826066 4550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:20.832940 4550 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a168ef653eff463bb88f94f6d019893f in term 1.
I20260925 15:07:20.834476 4549 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a [term 1 FOLLOWER]: Refusing update from remote peer a168ef653eff463bb88f94f6d019893f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:20.836447 4691 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:20.903746 4705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a168ef653eff463bb88f94f6d019893f. Latest consensus state: current_term: 1 leader_uuid: "a168ef653eff463bb88f94f6d019893f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } } }
I20260925 15:07:20.903774 4714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a168ef653eff463bb88f94f6d019893f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } } }
I20260925 15:07:20.904165 4705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:07:20.904269 4714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:07:20.922292 4722 catalog_manager.cc:1259] Loaded cluster ID: 1fd1c89b06894b4094f8651c3701e66c
I20260925 15:07:20.922381 4722 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a: loading cluster ID for follower catalog manager: success
I20260925 15:07:20.925139 4722 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a: acquiring CA information for follower catalog manager: success
I20260925 15:07:20.927073 4722 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b26119fa68e4478afb9f3cd5188fa8a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:07:21.212579 4691 sys_catalog.cc:455] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a168ef653eff463bb88f94f6d019893f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a168ef653eff463bb88f94f6d019893f" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39779 } } peers { permanent_uuid: "7b26119fa68e4478afb9f3cd5188fa8a" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 38475 } } peers { permanent_uuid: "8bf02686bc2d42c38cb2f943be5832d4" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 46543 } } }
I20260925 15:07:21.212944 4691 sys_catalog.cc:458] T 00000000000000000000000000000000 P a168ef653eff463bb88f94f6d019893f [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:21.613329 4689 catalog_manager.cc:1259] Loaded cluster ID: 1fd1c89b06894b4094f8651c3701e66c
I20260925 15:07:21.613430 4689 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4: loading cluster ID for follower catalog manager: success
I20260925 15:07:21.615743 4689 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4: acquiring CA information for follower catalog manager: success
I20260925 15:07:21.617470 4689 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8bf02686bc2d42c38cb2f943be5832d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:07:21.898042 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 4435
I20260925 15:07:21.899709 4497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:22.007684 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 4435
I20260925 15:07:22.024224 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 4509
I20260925 15:07:22.025261 4571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:22.130524 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 4509
I20260925 15:07:22.146140 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 4583
I20260925 15:07:22.147204 4645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:22.251763 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 4583
2026-09-25T15:07:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4326 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:22 dist-test-slave-nsrc krb5kdc[4730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:22 dist-test-slave-nsrc krb5kdc[4730](info): set up 2 sockets
Sep 25 15:07:22 dist-test-slave-nsrc krb5kdc[4730](info): commencing operation
krb5kdc: starting...
W20260925 15:07:23.256486 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:07:23 dist-test-slave-nsrc krb5kdc[4730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:07:23.463328 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:38301
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:38301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:23.773686 4743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:23.773976 4743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:23.774106 4743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:23.781136 4743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:23.781247 4743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:23.781322 4743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:23.781390 4743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:23.781450 4743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:23.791159 4743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:38301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:38301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:23.792428 4743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:23.793684 4743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:23.804698 4748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:23.804736 4749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:23.804757 4751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:23.805385 4743 server_base.cc:1034] running on GCE node
I20260925 15:07:23.806229 4743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:23.807556 4743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:23.808754 4743 hybrid_clock.cc:648] HybridClock initialized: now 1790348843808707 us; error 29 us; skew 500 ppm
I20260925 15:07:23.809093 4743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:23 dist-test-slave-nsrc krb5kdc[4730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348843, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:23.811877 4743 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:07:23.813277 4743 webserver.cc:460] Webserver started at http://127.22.236.62:39407/ using document root <none> and password file <none>
I20260925 15:07:23.813900 4743 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:23.814010 4743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:23.814282 4743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:23.817142 4743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "25c5f7c41e614268be48fda01b7d38b6"
format_stamp: "Formatted at 2026-09-25 15:07:23 on dist-test-slave-nsrc"
server_key: "da565a66dd19bf205c35b40ae8484f7b"
server_key_iv: "83131ec8bd513a03cd03fef427c2d646"
server_key_version: "encryptionkey@0"
I20260925 15:07:23.817829 4743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "25c5f7c41e614268be48fda01b7d38b6"
format_stamp: "Formatted at 2026-09-25 15:07:23 on dist-test-slave-nsrc"
server_key: "da565a66dd19bf205c35b40ae8484f7b"
server_key_iv: "83131ec8bd513a03cd03fef427c2d646"
server_key_version: "encryptionkey@0"
I20260925 15:07:23.822636 4743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:07:23.825838 4758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:23.827059 4743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:07:23.827210 4743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "25c5f7c41e614268be48fda01b7d38b6"
format_stamp: "Formatted at 2026-09-25 15:07:23 on dist-test-slave-nsrc"
server_key: "da565a66dd19bf205c35b40ae8484f7b"
server_key_iv: "83131ec8bd513a03cd03fef427c2d646"
server_key_version: "encryptionkey@0"
I20260925 15:07:23.827378 4743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:23.845168 4743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:23.849105 4743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:23.849349 4743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:23.866164 4743 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:38301
I20260925 15:07:23.866200 4809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:38301 every 8 connection(s)
I20260925 15:07:23.867439 4743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:23.871136 4810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:23.871126 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 4743
I20260925 15:07:23.871351 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:07:23.872181 23472 external_mini_cluster.cc:1428] Setting key f07c704cf733950a761f9e20c2626551
I20260925 15:07:23.884544 4810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6: Bootstrap starting.
Sep 25 15:07:23 dist-test-slave-nsrc krb5kdc[4730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:23.888087 4810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:23.889422 4810 log.cc:824] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:23.892649 4810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6: No bootstrap required, opened a new log
I20260925 15:07:23.896001 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:23.873932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.22.236.62:38301)
0925 15:07:23.874712 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:23.874724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:23.874763 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:07:23.877073 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:23.877096 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:23.877480 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:23.877804 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:23.879494 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:23.880817 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:23.882674 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:23.883305 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:23.883422 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:23.887947 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:23.887959 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:23.887982 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:23.888031 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:23.891672 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:23.892339 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:23.892347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:23.892355 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:23.892565 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:23.892924 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:23.892929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:23.892932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:23.893638 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:23.893921 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:23.894213 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:07:23.894676 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":352,"threads_started":1}
I20260925 15:07:23.899675 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:07:23.901710 4810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c5f7c41e614268be48fda01b7d38b6" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38301 } }
I20260925 15:07:23.902122 4810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:23.902230 4810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25c5f7c41e614268be48fda01b7d38b6, State: Initialized, Role: FOLLOWER
I20260925 15:07:23.902809 4810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c5f7c41e614268be48fda01b7d38b6" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38301 } }
I20260925 15:07:23.903030 4810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:23.903156 4810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:23.903302 4810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:23.905431 4810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c5f7c41e614268be48fda01b7d38b6" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38301 } }
I20260925 15:07:23.905869 4810 leader_election.cc:302] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25c5f7c41e614268be48fda01b7d38b6; no voters:
I20260925 15:07:23.906396 4810 leader_election.cc:288] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:23.906476 4815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:23.907078 4815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [term 1 LEADER]: Becoming Leader. State: Replica: 25c5f7c41e614268be48fda01b7d38b6, State: Running, Role: LEADER
I20260925 15:07:23.907536 4815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c5f7c41e614268be48fda01b7d38b6" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38301 } }
I20260925 15:07:23.908326 4810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:23.910166 4817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25c5f7c41e614268be48fda01b7d38b6. Latest consensus state: current_term: 1 leader_uuid: "25c5f7c41e614268be48fda01b7d38b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c5f7c41e614268be48fda01b7d38b6" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38301 } } }
I20260925 15:07:23.910545 4817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:23.910954 4816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25c5f7c41e614268be48fda01b7d38b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c5f7c41e614268be48fda01b7d38b6" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38301 } } }
I20260925 15:07:23.911284 4816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:23.912772 4824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:23.917021 4824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:23.924957 4824 catalog_manager.cc:1347] Generated new cluster ID: 139e479d8190454baeade9bb985f13ae
I20260925 15:07:23.925073 4824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:23.952170 4824 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:23.954066 4824 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:23.961520 4824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25c5f7c41e614268be48fda01b7d38b6: Generated new TSK 0
I20260925 15:07:23.962533 4824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:07:24.010324 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 4743
I20260925 15:07:24.011304 4805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:24.106739 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 4743
2026-09-25T15:07:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1852 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:24 dist-test-slave-nsrc krb5kdc[4838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:24 dist-test-slave-nsrc krb5kdc[4838](info): set up 2 sockets
Sep 25 15:07:24 dist-test-slave-nsrc krb5kdc[4838](info): commencing operation
krb5kdc: starting...
W20260925 15:07:25.108515 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:07:25 dist-test-slave-nsrc krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:07:25.317860 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:37191
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:37191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:25.629541 4851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:25.629833 4851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:25.629977 4851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:25.637094 4851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:25.637205 4851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:25.637283 4851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:25.637362 4851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:25.637427 4851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:25.647054 4851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:37191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:37191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:25.648417 4851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:25.649684 4851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:25.660868 4856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:25.660933 4857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:25.660892 4859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:25.661702 4851 server_base.cc:1034] running on GCE node
I20260925 15:07:25.662480 4851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:25.663810 4851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:25.665007 4851 hybrid_clock.cc:648] HybridClock initialized: now 1790348845664952 us; error 43 us; skew 500 ppm
I20260925 15:07:25.665346 4851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:25 dist-test-slave-nsrc krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348845, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:25.672236 4851 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:07:25.673623 4851 webserver.cc:460] Webserver started at http://127.22.236.62:45329/ using document root <none> and password file <none>
I20260925 15:07:25.674276 4851 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:25.674391 4851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:25.674664 4851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:25.677696 4851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01"
format_stamp: "Formatted at 2026-09-25 15:07:25 on dist-test-slave-nsrc"
server_key: "f326277dca769a123bfc3fc4b642c232"
server_key_iv: "b5dc5565645e1b6c38988284d9b4502d"
server_key_version: "encryptionkey@0"
I20260925 15:07:25.678401 4851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01"
format_stamp: "Formatted at 2026-09-25 15:07:25 on dist-test-slave-nsrc"
server_key: "f326277dca769a123bfc3fc4b642c232"
server_key_iv: "b5dc5565645e1b6c38988284d9b4502d"
server_key_version: "encryptionkey@0"
I20260925 15:07:25.683238 4851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 15:07:25.686403 4866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:25.687660 4851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:25.687812 4851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01"
format_stamp: "Formatted at 2026-09-25 15:07:25 on dist-test-slave-nsrc"
server_key: "f326277dca769a123bfc3fc4b642c232"
server_key_iv: "b5dc5565645e1b6c38988284d9b4502d"
server_key_version: "encryptionkey@0"
I20260925 15:07:25.687976 4851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:25.700006 4851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:25.703680 4851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:25.703922 4851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:25.719868 4851 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:37191
I20260925 15:07:25.719887 4917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:37191 every 8 connection(s)
I20260925 15:07:25.721230 4851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:25.724758 4918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:25.726509 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 4851
I20260925 15:07:25.726737 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:07:25.727564 23472 external_mini_cluster.cc:1428] Setting key d90c0d57e05cb03811d615ee9c68e818
I20260925 15:07:25.736747 4918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Bootstrap starting.
I20260925 15:07:25.740267 4918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:25.741461 4918 log.cc:824] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:07:25 dist-test-slave-nsrc krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:25.744306 4918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: No bootstrap required, opened a new log
I20260925 15:07:25.749341 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:25.729564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.22.236.62:37191)
0925 15:07:25.730294 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:25.730305 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:25.730348 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:07:25.731950 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:25.731971 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:25.732391 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:25.732692 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:25.734405 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:25.735665 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:25.737465 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:25.738118 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:25.738270 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:25.742648 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:25.742656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:25.742670 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:25.742708 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:25.745562 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:25.746185 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:25.746192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:25.746197 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:25.746358 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:25.746716 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:25.746720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:25.746722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:25.747245 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:25.747562 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:25.747934 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 15:07:25.748382 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":325,"threads_started":1}
I20260925 15:07:25.752393 4918 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } }
I20260925 15:07:25.752741 4918 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:25.752838 4918 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5bb7cbc833d49b3bdb37dbc7534bf01, State: Initialized, Role: FOLLOWER
I20260925 15:07:25.753322 4918 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } }
I20260925 15:07:25.753535 4918 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:25.753649 4918 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:25.753794 4918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:25.755816 4918 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } }
I20260925 15:07:25.756316 4918 leader_election.cc:302] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5bb7cbc833d49b3bdb37dbc7534bf01; no voters:
I20260925 15:07:25.756846 4918 leader_election.cc:288] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:25.756944 4923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:25.757915 4923 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 1 LEADER]: Becoming Leader. State: Replica: b5bb7cbc833d49b3bdb37dbc7534bf01, State: Running, Role: LEADER
I20260925 15:07:25.758443 4923 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } }
I20260925 15:07:25.758739 4918 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:25.760836 4924 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } } }
I20260925 15:07:25.761158 4924 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:25.761626 4925 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5bb7cbc833d49b3bdb37dbc7534bf01. Latest consensus state: current_term: 1 leader_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } } }
I20260925 15:07:25.761965 4925 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:25.768625 4930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:25.773886 4930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:25.781718 4930 catalog_manager.cc:1347] Generated new cluster ID: 865919df54e544b6b9d139d65754e1fe
I20260925 15:07:25.781826 4930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:25.796123 4930 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:25.797355 4930 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:25.808130 4930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Generated new TSK 0
I20260925 15:07:25.808945 4930 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:07:25.865563 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37191
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:26.197634 4945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:26.197906 4945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:26.198050 4945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:26.205070 4945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:26.205174 4945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:26.205332 4945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:07:26.214819 4945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:26.216017 4945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:26.217342 4945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:26.229142 4953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:26.229183 4951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:26.229204 4950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:26.229838 4945 server_base.cc:1034] running on GCE node
I20260925 15:07:26.230388 4945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:26.231261 4945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:26.232442 4945 hybrid_clock.cc:648] HybridClock initialized: now 1790348846232399 us; error 37 us; skew 500 ppm
I20260925 15:07:26.232733 4945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:26 dist-test-slave-nsrc krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348846, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:26.235615 4945 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:07:26.236956 4945 webserver.cc:460] Webserver started at http://127.22.236.1:35875/ using document root <none> and password file <none>
I20260925 15:07:26.237583 4945 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:26.237694 4945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:26.237952 4945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:26.240690 4945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "cd21e894bc2d48fd95d5ed25314f52f6"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-nsrc"
server_key: "7e515b424c8fa984df7103c45aec1bf1"
server_key_iv: "183d50e4738a30f8fdf9463bd626c304"
server_key_version: "encryptionkey@0"
I20260925 15:07:26.241417 4945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "cd21e894bc2d48fd95d5ed25314f52f6"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-nsrc"
server_key: "7e515b424c8fa984df7103c45aec1bf1"
server_key_iv: "183d50e4738a30f8fdf9463bd626c304"
server_key_version: "encryptionkey@0"
I20260925 15:07:26.246290 4945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:07:26.249605 4960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:26.250921 4945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:07:26.251065 4945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "cd21e894bc2d48fd95d5ed25314f52f6"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-nsrc"
server_key: "7e515b424c8fa984df7103c45aec1bf1"
server_key_iv: "183d50e4738a30f8fdf9463bd626c304"
server_key_version: "encryptionkey@0"
I20260925 15:07:26.251237 4945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:26.270797 4945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:26.274881 4945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:26.275134 4945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:26.276003 4945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:26.277359 4945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:26.277431 4945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:26.277509 4945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:26.277566 4945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:26.305889 4945 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:38461
I20260925 15:07:26.305955 5072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:38461 every 8 connection(s)
I20260925 15:07:26.307099 4945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:26.308449 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 4945
I20260925 15:07:26.308646 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:07:26.309237 23472 external_mini_cluster.cc:1428] Setting key 547b716866a583aef55b29ee70c631db
Sep 25 15:07:26 dist-test-slave-nsrc krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348846, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:26.327682 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:26.309602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:45675 (local address 127.22.236.62:37191)
0925 15:07:26.310015 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:26.310021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:26.311119 (+ 1098us) server_negotiation.cc:407] Connection header received
0925 15:07:26.313449 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:26.313453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:26.313544 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:26.313641 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:26.315428 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:26.316174 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:26.318172 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:26.318567 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:26.318661 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:26.322056 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:26.322062 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:26.322067 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:26.322104 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:26.324993 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:26.325614 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:26.325619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:26.325623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:26.325749 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:26.326076 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:26.326080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:26.326083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:26.326410 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:26.326565 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:26.327118 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 15:07:26.327334 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":191,"threads_started":1}
I20260925 15:07:26.328495 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:26.310185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:45675)
0925 15:07:26.310897 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:26.310949 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:26.313152 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:26.313875 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:26.313884 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:26.314516 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:26.315248 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:26.315273 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:26.316294 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:26.316300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:26.318013 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:26.318026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:26.318769 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:26.318775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:26.319042 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:26.319658 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:26.319679 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:26.321846 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:26.325113 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:26.325119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:26.325130 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:26.325508 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:26.325836 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:26.325839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:26.325841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:26.325996 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:26.326508 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:26.326513 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:26.326799 (+ 286us) client_negotiation.cc:769] Sending connection context
0925 15:07:26.327005 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 15:07:26.327358 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":278,"threads_started":1}
I20260925 15:07:26.329905 5073 heartbeater.cc:347] Connected to a master server at 127.22.236.62:37191
I20260925 15:07:26.330235 5073 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:26.331110 5073 heartbeater.cc:511] Master 127.22.236.62:37191 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
I20260925 15:07:26.333237 4883 ts_manager.cc:194] Registered new tserver with Master: cd21e894bc2d48fd95d5ed25314f52f6 (127.22.236.1:38461)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
I20260925 15:07:26.334736 4883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:45675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:07:26.356839 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37191
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:26.662607 5079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:26.662905 5079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:26.663054 5079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:26.670287 5079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:26.670418 5079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:26.670575 5079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:07:26.680387 5079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:26.681643 5079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:26.683099 5079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:26.695772 5084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:26.695756 5085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:26.695735 5087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:26.696684 5079 server_base.cc:1034] running on GCE node
I20260925 15:07:26.697237 5079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:26.698125 5079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:26.699348 5079 hybrid_clock.cc:648] HybridClock initialized: now 1790348846699271 us; error 50 us; skew 500 ppm
I20260925 15:07:26.699644 5079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:26 dist-test-slave-nsrc krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348846, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:26.702776 5079 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:07:26.704170 5079 webserver.cc:460] Webserver started at http://127.22.236.2:33073/ using document root <none> and password file <none>
I20260925 15:07:26.704793 5079 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:26.704898 5079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:26.705158 5079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:26.707870 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "d418614b0fbc4b278b47966ab6ad4e3a"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-nsrc"
server_key: "1635bed02e64f03fda8e4008c12f9f92"
server_key_iv: "a7bf98007b6ca62a2e83efb592d34371"
server_key_version: "encryptionkey@0"
I20260925 15:07:26.708652 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "d418614b0fbc4b278b47966ab6ad4e3a"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-nsrc"
server_key: "1635bed02e64f03fda8e4008c12f9f92"
server_key_iv: "a7bf98007b6ca62a2e83efb592d34371"
server_key_version: "encryptionkey@0"
I20260925 15:07:26.713459 5079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 15:07:26.716821 5094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:26.718124 5079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:07:26.718274 5079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "d418614b0fbc4b278b47966ab6ad4e3a"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-nsrc"
server_key: "1635bed02e64f03fda8e4008c12f9f92"
server_key_iv: "a7bf98007b6ca62a2e83efb592d34371"
server_key_version: "encryptionkey@0"
I20260925 15:07:26.718449 5079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:26.736317 5079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:26.740114 5079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:26.740396 5079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:26.741292 5079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:26.742543 5079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:26.742612 5079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:26.742703 5079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:26.742744 5079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:26.772264 5079 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:39413
I20260925 15:07:26.772318 5206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:39413 every 8 connection(s)
I20260925 15:07:26.773486 5079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:26.775306 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 5079
I20260925 15:07:26.775490 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:07:26.776022 23472 external_mini_cluster.cc:1428] Setting key 3c1f94fa044eda15f0a46a22eb05b5b8
Sep 25 15:07:26 dist-test-slave-nsrc krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348846, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:26.792615 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:26.776178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:36271 (local address 127.22.236.62:37191)
0925 15:07:26.776364 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:26.776369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:26.777552 (+ 1183us) server_negotiation.cc:407] Connection header received
0925 15:07:26.779371 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:26.779377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:26.779466 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:26.779546 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:26.781062 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:26.781788 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:26.783266 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:26.783664 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:26.783771 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:26.787199 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:26.787204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:26.787207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:26.787238 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:26.789657 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:26.790339 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:26.790342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:26.790345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:26.790467 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:26.790850 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:26.790854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:26.790856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:26.791111 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:26.791245 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:26.791975 (+ 730us) server_negotiation.cc:299] Negotiation successful
0925 15:07:26.792236 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:07:26.793490 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:26.776644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:36271)
0925 15:07:26.777402 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:26.777448 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:26.779112 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:26.779788 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:26.779796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:26.780338 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:26.780890 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:26.780907 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:26.781935 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:26.781939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:26.783129 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:26.783141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:26.783872 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:26.783879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:26.784100 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:26.784735 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:26.784759 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:26.786977 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:26.789798 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:26.789807 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:26.789825 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:26.790218 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:26.790566 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:26.790571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:26.790575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:26.790753 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:26.791229 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:26.791237 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:26.791621 (+ 384us) client_negotiation.cc:769] Sending connection context
0925 15:07:26.791848 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 15:07:26.792384 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":312,"threads_started":1}
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
I20260925 15:07:26.794888 5207 heartbeater.cc:347] Connected to a master server at 127.22.236.62:37191
I20260925 15:07:26.795197 5207 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:26.795897 5207 heartbeater.cc:511] Master 127.22.236.62:37191 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
I20260925 15:07:26.797295 4883 ts_manager.cc:194] Registered new tserver with Master: d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413)
I20260925 15:07:26.798075 4883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:36271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:07:26.821523 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37191
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:27.126235 5212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:27.126505 5212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:27.126636 5212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:27.133667 5212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:27.133775 5212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:27.133939 5212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:07:27.143296 5212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:27.144538 5212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:27.145877 5212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:27.158221 5217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:27.158149 5220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:27.158183 5218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:27.159039 5212 server_base.cc:1034] running on GCE node
I20260925 15:07:27.159658 5212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:27.160640 5212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:27.161839 5212 hybrid_clock.cc:648] HybridClock initialized: now 1790348847161784 us; error 48 us; skew 500 ppm
I20260925 15:07:27.162173 5212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:27 dist-test-slave-nsrc krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348847, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:27.165241 5212 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:07:27.166647 5212 webserver.cc:460] Webserver started at http://127.22.236.3:33903/ using document root <none> and password file <none>
I20260925 15:07:27.167292 5212 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:27.167390 5212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:27.167630 5212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:27.170557 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "d853cda79b1a410abd65083195ef4f3b"
format_stamp: "Formatted at 2026-09-25 15:07:27 on dist-test-slave-nsrc"
server_key: "a32137ee11139b0361f7a0c49e479141"
server_key_iv: "4caed2259640fa3a2eed5bd48fbc1020"
server_key_version: "encryptionkey@0"
I20260925 15:07:27.171267 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "d853cda79b1a410abd65083195ef4f3b"
format_stamp: "Formatted at 2026-09-25 15:07:27 on dist-test-slave-nsrc"
server_key: "a32137ee11139b0361f7a0c49e479141"
server_key_iv: "4caed2259640fa3a2eed5bd48fbc1020"
server_key_version: "encryptionkey@0"
I20260925 15:07:27.176044 5212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260925 15:07:27.179339 5227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:27.180749 5212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:27.180920 5212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "d853cda79b1a410abd65083195ef4f3b"
format_stamp: "Formatted at 2026-09-25 15:07:27 on dist-test-slave-nsrc"
server_key: "a32137ee11139b0361f7a0c49e479141"
server_key_iv: "4caed2259640fa3a2eed5bd48fbc1020"
server_key_version: "encryptionkey@0"
I20260925 15:07:27.181102 5212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:27.196372 5212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:27.200627 5212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:27.200906 5212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:27.201858 5212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:27.203222 5212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:27.203297 5212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:27.203410 5212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:27.203464 5212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:27.233158 5212 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:37745
I20260925 15:07:27.233193 5339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:37745 every 8 connection(s)
I20260925 15:07:27.234375 5212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:27.239571 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 5212
I20260925 15:07:27.239753 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:07:27.240294 23472 external_mini_cluster.cc:1428] Setting key 890b1dc43b39b1294bdd8aeeb46dbb6b
Sep 25 15:07:27 dist-test-slave-nsrc krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348847, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:27.252604 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.236755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:50985 (local address 127.22.236.62:37191)
0925 15:07:27.236914 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.236918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.238104 (+ 1186us) server_negotiation.cc:407] Connection header received
0925 15:07:27.240094 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.240098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.240203 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.240287 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.241924 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.242726 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.244295 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.244698 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.244791 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.247799 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.247804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.247807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:27.247837 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.250192 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:27.250812 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:27.250816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:27.250818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:27.250920 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:27.251246 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:27.251249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:27.251251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:27.251497 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.251601 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.252192 (+ 591us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.252360 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:07:27.253525 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.237248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:50985)
0925 15:07:27.237941 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.237978 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:27.239802 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:27.240559 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:27.240572 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:27.241121 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:27.241729 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:27.241754 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.242881 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.242885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:27.244181 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:27.244194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.244871 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.244876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:27.245066 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.245622 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:27.245642 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:27.247630 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:27.250315 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:27.250321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:27.250334 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:27.250727 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:27.250991 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:27.250994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:27.250996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:27.251169 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:27.251581 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:27.251586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:27.251879 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 15:07:27.252106 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 15:07:27.252516 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":264,"threads_started":1}
I20260925 15:07:27.255036 5340 heartbeater.cc:347] Connected to a master server at 127.22.236.62:37191
I20260925 15:07:27.255398 5340 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:27.256108 5340 heartbeater.cc:511] Master 127.22.236.62:37191 requested a full tablet report, sending...
I20260925 15:07:27.257406 4883 ts_manager.cc:194] Registered new tserver with Master: d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745)
I20260925 15:07:27.258173 4883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:50985
I20260925 15:07:27.267478 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:07:27.283439 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.271790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.22.236.62:37191)
0925 15:07:27.271941 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.271945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.272366 (+ 421us) server_negotiation.cc:407] Connection header received
0925 15:07:27.272530 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.272546 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.272608 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.272682 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.273976 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.274670 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.276073 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.276473 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.276566 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.278798 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.278802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.278805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:27.278831 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.281336 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:27.281936 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:27.281939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:27.281942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:27.282065 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:27.282332 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:27.282335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:27.282337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:27.282613 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.282788 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.283011 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.283169 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:07:27.287288 4883 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:27.294087 4883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:27.324530 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.305554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:37745 (local address 127.0.0.1:38124)
0925 15:07:27.316001 (+ 10447us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.316021 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:27.316181 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:27.317163 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:27.317171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:27.317198 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:27.317814 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:27.317828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.319689 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.319694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:27.321529 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:27.321543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.323570 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.323577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:27.323764 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.323787 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:07:27.323904 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 15:07:27.324103 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10329,"mutex_wait_us":40,"thread_start_us":10195,"threads_started":1}
I20260925 15:07:27.324553 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.305380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.22.236.1:38461)
0925 15:07:27.316424 (+ 11044us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.316432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.316454 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:07:27.316533 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.316539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.316696 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.316901 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:27.317979 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.319144 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.321937 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.323288 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.323414 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.323620 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.324050 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.324229 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10879,"thread_start_us":11007,"threads_started":1}
I20260925 15:07:27.324764 5355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.306176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.22.236.3:37745)
0925 15:07:27.315270 (+ 9094us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.315279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.316317 (+ 1038us) server_negotiation.cc:407] Connection header received
0925 15:07:27.316439 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.316447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.316707 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.316894 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:27.318040 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.319571 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.321710 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.323064 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.323185 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.323431 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.323967 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.324458 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8925,"thread_start_us":8772,"threads_started":1}
I20260925 15:07:27.325522 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.306845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.22.236.2:39413)
0925 15:07:27.315576 (+ 8731us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.315585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.317968 (+ 2383us) server_negotiation.cc:407] Connection header received
0925 15:07:27.318239 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.318247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.318475 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.318627 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:27.319460 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.320957 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.322955 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.324286 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.324408 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.324592 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.325092 (+ 500us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.325242 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8576,"thread_start_us":8434,"threads_started":1}
I20260925 15:07:27.325568 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.306421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:39413 (local address 127.0.0.1:42722)
0925 15:07:27.317865 (+ 11444us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.317884 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:27.318062 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:27.318858 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:27.318862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:27.318885 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:27.319208 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:27.319217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.321122 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.321128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:27.322829 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:27.322838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.324752 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.324759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:27.324933 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.324958 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 15:07:27.325033 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:07:27.325147 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11282,"mutex_wait_us":4,"thread_start_us":10996,"threads_started":1}
I20260925 15:07:27.325659 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.305150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38461 (local address 127.0.0.1:37102)
0925 15:07:27.306066 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.306110 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:27.306282 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:27.316949 (+ 10667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:27.316958 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:27.316995 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:27.317748 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:27.317770 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.319272 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.319277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:27.321776 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:27.321790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.323475 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.323480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:27.323715 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.323789 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 15:07:27.323905 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 15:07:27.325245 (+ 1340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":1206,"thread_start_us":519,"threads_started":1}
I20260925 15:07:27.329062 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 4845d2a43d2646148ecb37fba56cf02f (DEFAULT_TABLE table=test-table [id=1d8c460e1da844dd9be08c29fdf5355b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:27.329101 5008 tablet_service.cc:1467] Processing CreateTablet for tablet 4845d2a43d2646148ecb37fba56cf02f (DEFAULT_TABLE table=test-table [id=1d8c460e1da844dd9be08c29fdf5355b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:27.330245 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4845d2a43d2646148ecb37fba56cf02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:27.330427 5008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4845d2a43d2646148ecb37fba56cf02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:27.330452 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 4845d2a43d2646148ecb37fba56cf02f (DEFAULT_TABLE table=test-table [id=1d8c460e1da844dd9be08c29fdf5355b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:27.331670 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4845d2a43d2646148ecb37fba56cf02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:27.337577 5073 heartbeater.cc:503] Master 127.22.236.62:37191 was elected leader, sending a full tablet report...
I20260925 15:07:27.342573 5358 tablet_bootstrap.cc:492] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b: Bootstrap starting.
I20260925 15:07:27.343976 5359 tablet_bootstrap.cc:492] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6: Bootstrap starting.
I20260925 15:07:27.345958 5358 tablet_bootstrap.cc:654] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:27.347209 5358 log.cc:824] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:27.347280 5359 tablet_bootstrap.cc:654] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:27.348496 5359 log.cc:824] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:27.351050 5360 tablet_bootstrap.cc:492] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a: Bootstrap starting.
I20260925 15:07:27.351752 5359 tablet_bootstrap.cc:492] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6: No bootstrap required, opened a new log
I20260925 15:07:27.352049 5359 ts_tablet_manager.cc:1397] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 15:07:27.353840 5360 tablet_bootstrap.cc:654] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:27.354908 5360 log.cc:824] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:27.357060 5358 tablet_bootstrap.cc:492] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b: No bootstrap required, opened a new log
I20260925 15:07:27.357322 5358 ts_tablet_manager.cc:1397] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260925 15:07:27.362143 5360 tablet_bootstrap.cc:492] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a: No bootstrap required, opened a new log
I20260925 15:07:27.362370 5360 ts_tablet_manager.cc:1397] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260925 15:07:27.362351 5359 raft_consensus.cc:348] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } }
I20260925 15:07:27.362784 5359 raft_consensus.cc:374] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:27.362879 5359 raft_consensus.cc:729] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd21e894bc2d48fd95d5ed25314f52f6, State: Initialized, Role: FOLLOWER
I20260925 15:07:27.363579 5359 consensus_queue.cc:260] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } }
I20260925 15:07:27.368134 5359 ts_tablet_manager.cc:1428] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260925 15:07:27.369309 5358 raft_consensus.cc:348] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } }
I20260925 15:07:27.369721 5358 raft_consensus.cc:374] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:27.369830 5358 raft_consensus.cc:729] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d853cda79b1a410abd65083195ef4f3b, State: Initialized, Role: FOLLOWER
I20260925 15:07:27.370133 5360 raft_consensus.cc:348] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } }
I20260925 15:07:27.370505 5360 raft_consensus.cc:374] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:27.370608 5360 raft_consensus.cc:729] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d418614b0fbc4b278b47966ab6ad4e3a, State: Initialized, Role: FOLLOWER
I20260925 15:07:27.370471 5358 consensus_queue.cc:260] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } }
I20260925 15:07:27.371167 5360 consensus_queue.cc:260] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } }
I20260925 15:07:27.372291 5358 ts_tablet_manager.cc:1428] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b: Time spent starting tablet: real 0.015s user 0.004s sys 0.010s
I20260925 15:07:27.372541 5207 heartbeater.cc:503] Master 127.22.236.62:37191 was elected leader, sending a full tablet report...
I20260925 15:07:27.372753 5340 heartbeater.cc:503] Master 127.22.236.62:37191 was elected leader, sending a full tablet report...
I20260925 15:07:27.373805 5360 ts_tablet_manager.cc:1428] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20260925 15:07:27.475963 5365 raft_consensus.cc:482] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:27.476277 5365 raft_consensus.cc:504] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } }
I20260925 15:07:27.478065 5365 leader_election.cc:288] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd21e894bc2d48fd95d5ed25314f52f6 (127.22.236.1:38461), d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413)
I20260925 15:07:27.484400 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.478370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38461 (local address 127.22.236.3:44479)
0925 15:07:27.478570 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.478589 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:27.478715 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:27.479226 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:27.479231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:27.479262 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:27.479448 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:27.479460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.480746 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.480749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:27.482500 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:27.482516 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.483793 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.483797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:27.483928 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.483953 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 15:07:27.484019 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:07:27.484113 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":46}
I20260925 15:07:27.484498 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.478387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:44479 (local address 127.22.236.1:38461)
0925 15:07:27.478567 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.478572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.478649 (+ 77us) server_negotiation.cc:407] Connection header received
0925 15:07:27.478866 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.478872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.478950 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.479039 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:27.479613 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.480625 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.482674 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.483655 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.483800 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.483875 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.484067 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.484216 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:07:27.485400 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.478766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:39413 (local address 127.22.236.3:59367)
0925 15:07:27.479487 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.479503 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:27.479656 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:27.479962 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:27.479964 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:27.479980 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:27.480278 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:27.480287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.481330 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.481335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:27.483047 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:27.483058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.484690 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.484696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:27.484876 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.484918 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:07:27.485008 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:07:27.485136 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":1,"thread_start_us":365,"threads_started":1}
I20260925 15:07:27.485404 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.478884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:59367 (local address 127.22.236.2:39413)
0925 15:07:27.479109 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.479116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.479570 (+ 454us) server_negotiation.cc:407] Connection header received
0925 15:07:27.479781 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.479787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.479861 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.479960 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:27.480412 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.481190 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.483192 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.484328 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.484470 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.484539 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.485044 (+ 505us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.485176 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:07:27.485595 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4845d2a43d2646148ecb37fba56cf02f" candidate_uuid: "d853cda79b1a410abd65083195ef4f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" is_pre_election: true
W20260925 15:07:27.485790 5341 tablet.cc:2369] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:07:27.486194 5028 raft_consensus.cc:2454] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d853cda79b1a410abd65083195ef4f3b in term 0.
I20260925 15:07:27.486783 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4845d2a43d2646148ecb37fba56cf02f" candidate_uuid: "d853cda79b1a410abd65083195ef4f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" is_pre_election: true
I20260925 15:07:27.486886 5230 leader_election.cc:302] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd21e894bc2d48fd95d5ed25314f52f6, d853cda79b1a410abd65083195ef4f3b; no voters:
I20260925 15:07:27.487282 5365 raft_consensus.cc:2781] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:27.487287 5162 raft_consensus.cc:2454] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d853cda79b1a410abd65083195ef4f3b in term 0.
I20260925 15:07:27.487408 5365 raft_consensus.cc:482] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:27.487496 5365 raft_consensus.cc:3037] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:27.489734 5365 raft_consensus.cc:504] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } }
I20260925 15:07:27.490304 5365 leader_election.cc:288] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [CANDIDATE]: Term 1 election: Requested vote from peers cd21e894bc2d48fd95d5ed25314f52f6 (127.22.236.1:38461), d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413)
I20260925 15:07:27.490746 5028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4845d2a43d2646148ecb37fba56cf02f" candidate_uuid: "d853cda79b1a410abd65083195ef4f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd21e894bc2d48fd95d5ed25314f52f6"
I20260925 15:07:27.490772 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4845d2a43d2646148ecb37fba56cf02f" candidate_uuid: "d853cda79b1a410abd65083195ef4f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d418614b0fbc4b278b47966ab6ad4e3a"
I20260925 15:07:27.490960 5162 raft_consensus.cc:3037] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:27.490960 5028 raft_consensus.cc:3037] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:27.494073 5162 raft_consensus.cc:2454] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d853cda79b1a410abd65083195ef4f3b in term 1.
I20260925 15:07:27.494079 5028 raft_consensus.cc:2454] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d853cda79b1a410abd65083195ef4f3b in term 1.
I20260925 15:07:27.494452 5230 leader_election.cc:302] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d418614b0fbc4b278b47966ab6ad4e3a, d853cda79b1a410abd65083195ef4f3b; no voters:
I20260925 15:07:27.494691 5365 raft_consensus.cc:2781] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:27.495111 5365 raft_consensus.cc:686] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [term 1 LEADER]: Becoming Leader. State: Replica: d853cda79b1a410abd65083195ef4f3b, State: Running, Role: LEADER
I20260925 15:07:27.495565 5365 consensus_queue.cc:237] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } }
I20260925 15:07:27.499835 4881 catalog_manager.cc:5495] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b reported cstate change: term changed from 0 to 1, leader changed from <none> to d853cda79b1a410abd65083195ef4f3b (127.22.236.3). New cstate: current_term: 1 leader_uuid: "d853cda79b1a410abd65083195ef4f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } health_report { overall_health: UNKNOWN } } }
W20260925 15:07:27.525223 5208 tablet.cc:2369] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:07:27.526400 5355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.521158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38132 (local address 127.22.236.3:37745)
0925 15:07:27.521331 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.521337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.521371 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:07:27.521571 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.521575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.521653 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.521775 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:27.522164 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.522894 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.524311 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.524709 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.524816 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.525317 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:27.525844 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:27.525984 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.526073 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.526154 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:07:27.533954 5162 raft_consensus.cc:1261] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a [term 1 FOLLOWER]: Refusing update from remote peer d853cda79b1a410abd65083195ef4f3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:27.534003 5028 raft_consensus.cc:1261] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Refusing update from remote peer d853cda79b1a410abd65083195ef4f3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:27.534868 5365 consensus_queue.cc:1035] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:27.535185 5368 consensus_queue.cc:1035] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:27.545778 5371 mvcc.cc:205] Tried to move back new op lower bound from 7333268879489843200 to 7333268879345643520. Current Snapshot: MvccSnapshot[applied={T|T < 7333268879489843200}]
I20260925 15:07:27.559178 4882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:27.559545 4882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:27.563687 4882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Sending DeleteTablet for 3 replicas of tablet 4845d2a43d2646148ecb37fba56cf02f
I20260925 15:07:27.564945 5275 tablet_service.cc:1514] Processing DeleteTablet for tablet 4845d2a43d2646148ecb37fba56cf02f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:27 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:38124
I20260925 15:07:27.565019 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 4845d2a43d2646148ecb37fba56cf02f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:27 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:42722
I20260925 15:07:27.565863 5381 tablet_replica.cc:330] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b: stopping tablet replica
I20260925 15:07:27.566009 5008 tablet_service.cc:1514] Processing DeleteTablet for tablet 4845d2a43d2646148ecb37fba56cf02f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:27 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:37102
I20260925 15:07:27.566437 5381 raft_consensus.cc:2229] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:27.566498 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 4945
I20260925 15:07:27.573915 5381 raft_consensus.cc:2258] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:27.577239 5382 tablet_replica.cc:330] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a: stopping tablet replica
I20260925 15:07:27.577703 5382 raft_consensus.cc:2229] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:27.578217 5382 raft_consensus.cc:2258] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:27.579618 5381 ts_tablet_manager.cc:1905] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:27.580193 5382 ts_tablet_manager.cc:1905] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:27.581655 5383 tablet_replica.cc:330] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6: stopping tablet replica
I20260925 15:07:27.582118 5383 raft_consensus.cc:2229] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:27.582644 5383 raft_consensus.cc:2258] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:27.584740 5383 ts_tablet_manager.cc:1905] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:27.587199 5382 ts_tablet_manager.cc:1918] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:27.587360 5382 log.cc:1196] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/4845d2a43d2646148ecb37fba56cf02f
I20260925 15:07:27.587864 5382 ts_tablet_manager.cc:1939] T 4845d2a43d2646148ecb37fba56cf02f P d418614b0fbc4b278b47966ab6ad4e3a: Deleting consensus metadata
I20260925 15:07:27.589754 4869 catalog_manager.cc:4841] TS d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413): tablet 4845d2a43d2646148ecb37fba56cf02f (table test-table [id=1d8c460e1da844dd9be08c29fdf5355b]) successfully deleted
I20260925 15:07:27.591415 5383 ts_tablet_manager.cc:1918] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:27.591576 5383 log.cc:1196] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/4845d2a43d2646148ecb37fba56cf02f
I20260925 15:07:27.592183 5383 ts_tablet_manager.cc:1939] T 4845d2a43d2646148ecb37fba56cf02f P cd21e894bc2d48fd95d5ed25314f52f6: Deleting consensus metadata
I20260925 15:07:27.593253 5381 ts_tablet_manager.cc:1918] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:27.593439 5381 log.cc:1196] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/4845d2a43d2646148ecb37fba56cf02f
I20260925 15:07:27.594009 5381 ts_tablet_manager.cc:1939] T 4845d2a43d2646148ecb37fba56cf02f P d853cda79b1a410abd65083195ef4f3b: Deleting consensus metadata
I20260925 15:07:27.594070 4869 catalog_manager.cc:4841] TS cd21e894bc2d48fd95d5ed25314f52f6 (127.22.236.1:38461): tablet 4845d2a43d2646148ecb37fba56cf02f (table test-table [id=1d8c460e1da844dd9be08c29fdf5355b]) successfully deleted
I20260925 15:07:27.595371 4868 catalog_manager.cc:4841] TS d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745): tablet 4845d2a43d2646148ecb37fba56cf02f (table test-table [id=1d8c460e1da844dd9be08c29fdf5355b]) successfully deleted
Sep 25 15:07:27 dist-test-slave-nsrc krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:07:27.608619 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.567796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.22.236.1:38461)
0925 15:07:27.567979 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.567986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.584881 (+ 16895us) server_negotiation.cc:407] Connection header received
0925 15:07:27.587699 (+ 2818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.587706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.587781 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.587864 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.591630 (+ 3766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.592891 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.596738 (+ 3847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.597358 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.597505 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.602103 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.602121 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.602142 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:27.602189 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.605850 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:27.606495 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:27.606505 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:27.606509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:27.606661 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:27.606984 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:27.606989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:27.606993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:27.607467 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.607770 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.607933 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.608229 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:07:27.609396 5068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:27.697088 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 4945
I20260925 15:07:27.719094 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 5079
Sep 25 15:07:27 dist-test-slave-nsrc krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:07:27.734283 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.720200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42728 (local address 127.22.236.2:39413)
0925 15:07:27.720363 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.720367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.720382 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:07:27.720432 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.720436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.720493 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.720585 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.721837 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.722637 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.723968 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.724423 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.724535 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.728563 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.728583 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.728606 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:27.728658 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.731349 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:27.732100 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:27.732108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:27.732110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:27.732281 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:27.732639 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:27.732644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:27.732648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:27.733048 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.733318 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.733636 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.733837 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:07:27.734937 5202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:27.822615 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 5079
I20260925 15:07:27.836598 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 5212
Sep 25 15:07:27 dist-test-slave-nsrc krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:07:27.854265 5355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.837331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38140 (local address 127.22.236.3:37745)
0925 15:07:27.837499 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.837503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.837545 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:07:27.837743 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.837747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.837807 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.837944 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.839664 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.840572 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.842413 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:27.842861 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:27.842951 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:27.848390 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.848402 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.848417 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:27.848456 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.851202 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:27.852018 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:27.852025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:27.852027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:27.852400 (+ 373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:27.852791 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:27.852794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:27.852796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:27.853178 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.853396 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.853772 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.853934 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:07:27.854988 5335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:27.972904 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 5212
I20260925 15:07:27.987869 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 4851
I20260925 15:07:27.988960 4913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:28.099066 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 4851
I20260925 15:07:28.116761 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:37191
--webserver_interface=127.22.236.62
--webserver_port=45329
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:37191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:28.431998 5389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:28.432289 5389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:28.432425 5389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:28.439563 5389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:28.439666 5389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:28.439738 5389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:28.439811 5389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:28.439880 5389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:28.449620 5389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:37191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:37191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=45329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:28.450876 5389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:28.452124 5389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:28.472998 5398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:28.472998 5396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:28.473299 5395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:28.473680 5389 server_base.cc:1034] running on GCE node
I20260925 15:07:28.474526 5389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:28.476082 5389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:28.477335 5389 hybrid_clock.cc:648] HybridClock initialized: now 1790348848477279 us; error 45 us; skew 500 ppm
I20260925 15:07:28.477731 5389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:28 dist-test-slave-nsrc krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348848, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:28.481154 5389 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:07:28.482810 5389 webserver.cc:460] Webserver started at http://127.22.236.62:45329/ using document root <none> and password file <none>
I20260925 15:07:28.483549 5389 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:28.483678 5389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:28.489576 5389 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 15:07:28.492697 5405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:28.494213 5389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 15:07:28.494400 5389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01"
format_stamp: "Formatted at 2026-09-25 15:07:25 on dist-test-slave-nsrc"
server_key: "f326277dca769a123bfc3fc4b642c232"
server_key_iv: "b5dc5565645e1b6c38988284d9b4502d"
server_key_version: "encryptionkey@0"
I20260925 15:07:28.495529 5389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:28.515440 5389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:28.519652 5389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:28.519935 5389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:28.538235 5389 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:37191
I20260925 15:07:28.538267 5456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:37191 every 8 connection(s)
I20260925 15:07:28.539827 5389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:28.545701 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 5389
I20260925 15:07:28.546707 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:38461
--local_ip_for_outbound_sockets=127.22.236.1
--tserver_master_addrs=127.22.236.62:37191
--webserver_port=35875
--webserver_interface=127.22.236.1
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:07:28.547752 5457 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:07:28.552465 5457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Bootstrap starting.
I20260925 15:07:28.572750 5457 log.cc:824] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:28.585060 5457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:07:28.585584 5457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Bootstrap complete.
I20260925 15:07:28.594731 5457 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } }
I20260925 15:07:28.595599 5457 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5bb7cbc833d49b3bdb37dbc7534bf01, State: Initialized, Role: FOLLOWER
I20260925 15:07:28.596288 5457 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } }
I20260925 15:07:28.596524 5457 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:28.596642 5457 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:28.596793 5457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:07:28.600951 5457 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } }
I20260925 15:07:28.601459 5457 leader_election.cc:302] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5bb7cbc833d49b3bdb37dbc7534bf01; no voters:
I20260925 15:07:28.602088 5457 leader_election.cc:288] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:07:28.602211 5461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:07:28.604307 5457 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:28.608716 5461 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 2 LEADER]: Becoming Leader. State: Replica: b5bb7cbc833d49b3bdb37dbc7534bf01, State: Running, Role: LEADER
I20260925 15:07:28.609367 5461 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } }
I20260925 15:07:28.613560 5462 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } } }
I20260925 15:07:28.613987 5462 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:28.614360 5463 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5bb7cbc833d49b3bdb37dbc7534bf01. Latest consensus state: current_term: 2 leader_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } } }
I20260925 15:07:28.614630 5463 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:28.627197 5471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:28.636055 5471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:28.637506 5471 catalog_manager.cc:1259] Loaded cluster ID: 865919df54e544b6b9d139d65754e1fe
I20260925 15:07:28.637687 5471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:28.640755 5471 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 15:07:28.657248 5471 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 15:07:28.666330 5471 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:28.668023 5471 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Loaded TSK: 0
I20260925 15:07:28.669600 5471 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:07:28.885308 5459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:28.885573 5459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:28.885694 5459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:28.892832 5459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:28.892931 5459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:28.893065 5459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:07:28.902781 5459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:38461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=35875
--tserver_master_addrs=127.22.236.62:37191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:28.904079 5459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:28.905452 5459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:28.919381 5485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:28.919355 5486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:28.920095 5488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:28.920586 5459 server_base.cc:1034] running on GCE node
I20260925 15:07:28.921185 5459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:28.922159 5459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:28.923369 5459 hybrid_clock.cc:648] HybridClock initialized: now 1790348848923327 us; error 39 us; skew 500 ppm
I20260925 15:07:28.923694 5459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:28 dist-test-slave-nsrc krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348848, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:28.926669 5459 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:07:28.928253 5459 webserver.cc:460] Webserver started at http://127.22.236.1:35875/ using document root <none> and password file <none>
I20260925 15:07:28.928894 5459 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:28.929011 5459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:28.934128 5459 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 15:07:28.936739 5495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:28.938076 5459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:28.938221 5459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "cd21e894bc2d48fd95d5ed25314f52f6"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-nsrc"
server_key: "7e515b424c8fa984df7103c45aec1bf1"
server_key_iv: "183d50e4738a30f8fdf9463bd626c304"
server_key_version: "encryptionkey@0"
I20260925 15:07:28.939138 5459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:28.967662 5459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:28.971935 5459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:28.972201 5459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:28.973130 5459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:28.974366 5459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:28.974459 5459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:28.974561 5459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:28.974632 5459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:29.003718 5459 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:38461
I20260925 15:07:29.003775 5607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:38461 every 8 connection(s)
I20260925 15:07:29.005317 5459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:29.010335 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 5459
I20260925 15:07:29.010947 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:39413
--local_ip_for_outbound_sockets=127.22.236.2
--tserver_master_addrs=127.22.236.62:37191
--webserver_port=33073
--webserver_interface=127.22.236.2
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:07:29 dist-test-slave-nsrc krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348848, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:29.029572 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.008224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:48887 (local address 127.22.236.62:37191)
0925 15:07:29.008767 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:29.008779 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:29.009340 (+ 561us) server_negotiation.cc:407] Connection header received
0925 15:07:29.011639 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:29.011657 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:29.012134 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:29.012576 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:29.013965 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.015775 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.017338 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.018014 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.018151 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.022108 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:29.022116 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:29.022131 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:29.022169 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:29.025228 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:29.026086 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:29.026093 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:29.026098 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:29.026291 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:29.026788 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:29.026792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:29.026794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:29.027256 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:29.027400 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:29.028351 (+ 951us) server_negotiation.cc:299] Negotiation successful
0925 15:07:29.028652 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":222,"threads_started":1}
I20260925 15:07:29.029969 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.008400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:48887)
0925 15:07:29.009161 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:29.009208 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:29.011014 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:29.012688 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:29.012696 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:29.013183 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:29.013728 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.013745 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.015939 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.015943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.017195 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.017207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.018239 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.018247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.018530 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.019244 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:29.019273 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:29.021847 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:29.025367 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:29.025376 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:29.025392 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:29.025965 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:29.026416 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:29.026421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:29.026424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:29.026674 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:29.027387 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:29.027395 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:29.027820 (+ 425us) client_negotiation.cc:769] Sending connection context
0925 15:07:29.028167 (+ 347us) client_negotiation.cc:240] Negotiation successful
0925 15:07:29.028722 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":297,"threads_started":1}
I20260925 15:07:29.032725 5608 heartbeater.cc:347] Connected to a master server at 127.22.236.62:37191
I20260925 15:07:29.033020 5608 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:29.033736 5608 heartbeater.cc:511] Master 127.22.236.62:37191 requested a full tablet report, sending...
I20260925 15:07:29.035810 5422 ts_manager.cc:194] Registered new tserver with Master: cd21e894bc2d48fd95d5ed25314f52f6 (127.22.236.1:38461)
I20260925 15:07:29.037289 5422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:48887
W20260925 15:07:29.327641 5612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:29.327910 5612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:29.328058 5612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:29.335144 5612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:29.335243 5612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:29.335378 5612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:07:29.345041 5612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:39413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=33073
--tserver_master_addrs=127.22.236.62:37191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:29.346226 5612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:29.347479 5612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:29.359488 5617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:29.359557 5618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:29.359539 5620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:29.360554 5612 server_base.cc:1034] running on GCE node
I20260925 15:07:29.361114 5612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:29.362020 5612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:29.363211 5612 hybrid_clock.cc:648] HybridClock initialized: now 1790348849363160 us; error 47 us; skew 500 ppm
I20260925 15:07:29.363500 5612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:29 dist-test-slave-nsrc krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348849, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:29.366425 5612 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:07:29.367810 5612 webserver.cc:460] Webserver started at http://127.22.236.2:33073/ using document root <none> and password file <none>
I20260925 15:07:29.368495 5612 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:29.368623 5612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:29.373832 5612 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:07:29.376494 5627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:29.377817 5612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:07:29.377969 5612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "d418614b0fbc4b278b47966ab6ad4e3a"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-nsrc"
server_key: "1635bed02e64f03fda8e4008c12f9f92"
server_key_iv: "a7bf98007b6ca62a2e83efb592d34371"
server_key_version: "encryptionkey@0"
I20260925 15:07:29.378911 5612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:29.393738 5612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:29.397795 5612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:29.398051 5612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:29.399055 5612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:29.400511 5612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:29.400637 5612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:29.400741 5612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:29.400811 5612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:29.430512 5612 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:39413
I20260925 15:07:29.430583 5739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:39413 every 8 connection(s)
I20260925 15:07:29.431906 5612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:29.433454 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 5612
I20260925 15:07:29.434129 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:37745
--local_ip_for_outbound_sockets=127.22.236.3
--tserver_master_addrs=127.22.236.62:37191
--webserver_port=33903
--webserver_interface=127.22.236.3
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:07:29 dist-test-slave-nsrc krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348849, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:29.453377 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.434615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:47795 (local address 127.22.236.62:37191)
0925 15:07:29.434845 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:29.434851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:29.436009 (+ 1158us) server_negotiation.cc:407] Connection header received
0925 15:07:29.438426 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:29.438431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:29.438503 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:29.438590 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:29.440459 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.441322 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.443319 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.443960 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.444079 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.447608 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:29.447613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:29.447616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:29.447652 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:29.450117 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:29.450977 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:29.450982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:29.450984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:29.451109 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:29.451597 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:29.451601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:29.451603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:29.451908 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:29.452025 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:29.452954 (+ 929us) server_negotiation.cc:299] Negotiation successful
0925 15:07:29.453135 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:07:29.454761 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.435077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:47795)
0925 15:07:29.435847 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:29.435898 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:29.438109 (+ 2211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:29.438866 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:29.438877 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:29.439526 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:29.440312 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.440334 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.441492 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.441497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.443184 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.443196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.444193 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.444200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.444407 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.445099 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:29.445121 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:29.447381 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:29.450248 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:29.450258 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:29.450274 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:29.450846 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:29.451210 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:29.451215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:29.451219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:29.451462 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:29.452038 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:29.452046 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:29.452498 (+ 452us) client_negotiation.cc:769] Sending connection context
0925 15:07:29.452810 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 15:07:29.453368 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":275,"threads_started":1}
I20260925 15:07:29.456269 5740 heartbeater.cc:347] Connected to a master server at 127.22.236.62:37191
I20260925 15:07:29.456596 5740 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:29.457296 5740 heartbeater.cc:511] Master 127.22.236.62:37191 requested a full tablet report, sending...
I20260925 15:07:29.458642 5422 ts_manager.cc:194] Registered new tserver with Master: d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413)
I20260925 15:07:29.459421 5422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:47795
W20260925 15:07:29.742867 5743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:29.743156 5743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:29.743309 5743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:29.750394 5743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:29.750514 5743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:29.750687 5743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:07:29.760070 5743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:37745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=33903
--tserver_master_addrs=127.22.236.62:37191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:29.761312 5743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:29.762586 5743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:29.774871 5749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:29.774850 5751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:29.775110 5748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:29.775759 5743 server_base.cc:1034] running on GCE node
I20260925 15:07:29.776409 5743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:29.777354 5743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:29.778554 5743 hybrid_clock.cc:648] HybridClock initialized: now 1790348849778507 us; error 44 us; skew 500 ppm
I20260925 15:07:29.778848 5743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:29 dist-test-slave-nsrc krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348849, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:29.781780 5743 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:07:29.783268 5743 webserver.cc:460] Webserver started at http://127.22.236.3:33903/ using document root <none> and password file <none>
I20260925 15:07:29.783913 5743 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:29.784039 5743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:29.789127 5743 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:29.791613 5758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:29.792872 5743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:07:29.793009 5743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "d853cda79b1a410abd65083195ef4f3b"
format_stamp: "Formatted at 2026-09-25 15:07:27 on dist-test-slave-nsrc"
server_key: "a32137ee11139b0361f7a0c49e479141"
server_key_iv: "4caed2259640fa3a2eed5bd48fbc1020"
server_key_version: "encryptionkey@0"
I20260925 15:07:29.793907 5743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:29.813465 5743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:29.817198 5743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:29.817441 5743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:29.818413 5743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:29.819769 5743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:29.819872 5743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:29.819979 5743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:29.820053 5743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:29.849607 5743 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:37745
I20260925 15:07:29.849674 5870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:37745 every 8 connection(s)
I20260925 15:07:29.850977 5743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:29.856684 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 5743
Sep 25 15:07:29 dist-test-slave-nsrc krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348849, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:29.871666 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.853328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:36249 (local address 127.22.236.62:37191)
0925 15:07:29.853490 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:29.853495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:29.854587 (+ 1092us) server_negotiation.cc:407] Connection header received
0925 15:07:29.856424 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:29.856428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:29.856497 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:29.856594 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:29.858424 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.859323 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.861303 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.861782 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.861865 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.865738 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:29.865743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:29.865746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:29.865776 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:29.868521 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:29.869363 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:29.869368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:29.869371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:29.869559 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:29.870005 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:29.870010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:29.870014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:29.870291 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:29.870416 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:29.871105 (+ 689us) server_negotiation.cc:299] Negotiation successful
0925 15:07:29.871324 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":51}
I20260925 15:07:29.871902 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.857630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.22.236.62:37191)
0925 15:07:29.858117 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:29.858123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:29.858573 (+ 450us) server_negotiation.cc:407] Connection header received
0925 15:07:29.858696 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:29.858701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:29.858771 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:29.858872 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:29.860589 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.861349 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.862941 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.863371 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.863458 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.866347 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:29.866352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:29.866355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:29.866407 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:29.869236 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:29.869973 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:29.869978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:29.869981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:29.870095 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:29.870421 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:29.870425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:29.870428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:29.870673 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:29.870905 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:29.871177 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 15:07:29.871470 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":373,"thread_start_us":252,"threads_started":1}
I20260925 15:07:29.872985 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.853769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:36249)
0925 15:07:29.854430 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:29.854475 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:29.856095 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:29.856865 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:29.856876 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:29.857540 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:29.858249 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.858276 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.859491 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.859497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.861174 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.861187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.862001 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.862009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.862285 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.862979 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:29.863009 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:29.865523 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:29.868657 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:29.868667 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:29.868686 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:29.869255 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:29.869680 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:29.869684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:29.869687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:29.869885 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:29.870417 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:29.870425 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:29.870750 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 15:07:29.870983 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 15:07:29.871514 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":245,"threads_started":1}
I20260925 15:07:29.874348 5871 heartbeater.cc:347] Connected to a master server at 127.22.236.62:37191
I20260925 15:07:29.874629 5871 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:29.875316 5871 heartbeater.cc:511] Master 127.22.236.62:37191 requested a full tablet report, sending...
I20260925 15:07:29.876744 5422 ts_manager.cc:194] Registered new tserver with Master: d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745)
I20260925 15:07:29.877588 5422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:36249
I20260925 15:07:29.886567 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:07:29.892762 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.887382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.22.236.62:37191)
0925 15:07:29.887581 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:29.887587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:29.887964 (+ 377us) server_negotiation.cc:407] Connection header received
0925 15:07:29.888167 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:29.888171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:29.888312 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:29.888433 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:29.888783 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.889461 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.890900 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.891327 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.891418 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.891803 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:29.892316 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:29.892421 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:29.892459 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 15:07:29.892519 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:07:29.893544 5422 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:29.900578 5422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:29.922430 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.911877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:39413 (local address 127.0.0.1:42738)
0925 15:07:29.912808 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:29.912847 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:29.913013 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:29.914882 (+ 1869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:29.914887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:29.914906 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:29.915261 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.915275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.917079 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.917084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.919340 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.919354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.921493 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.921499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.921701 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.921749 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:07:29.921874 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 15:07:29.922054 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":1420,"threads_started":1}
I20260925 15:07:29.923350 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.912361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.22.236.2:39413)
0925 15:07:29.914041 (+ 1680us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:29.914050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:29.914085 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:07:29.914176 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:29.914183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:29.914412 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:29.914607 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:29.915447 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.916916 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.919502 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.921012 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.921131 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.921359 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:29.922852 (+ 1493us) server_negotiation.cc:299] Negotiation successful
0925 15:07:29.923019 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1518,"thread_start_us":1633,"threads_started":1}
I20260925 15:07:29.924698 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.912078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:37745 (local address 127.0.0.1:38144)
0925 15:07:29.913413 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:29.913430 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:29.913528 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:29.914637 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:29.914643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:29.914680 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:29.916087 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.916101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.918221 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.918226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.921177 (+ 2951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.921191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.923993 (+ 2802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.924002 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.924198 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.924225 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 15:07:29.924274 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:07:29.924383 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1218,"thread_start_us":1113,"threads_started":1}
I20260925 15:07:29.925767 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.912772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.22.236.3:37745)
0925 15:07:29.914047 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:29.914055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:29.914084 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:07:29.914177 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:29.914183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:29.914412 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:29.916307 (+ 1895us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:29.916570 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.918049 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.921406 (+ 3357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.923771 (+ 2365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.924930 (+ 1159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.925192 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:29.925277 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 15:07:29.925437 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1114,"thread_start_us":1721,"threads_started":1}
I20260925 15:07:29.928270 5675 tablet_service.cc:1467] Processing CreateTablet for tablet 4c5e6e3133b841a2ae1d9399e52f13c4 (DEFAULT_TABLE table=test-table [id=fd7be3c5a0bd46a7868f7d3092b83063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:29.929612 5675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c5e6e3133b841a2ae1d9399e52f13c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:29.929319 5806 tablet_service.cc:1467] Processing CreateTablet for tablet 4c5e6e3133b841a2ae1d9399e52f13c4 (DEFAULT_TABLE table=test-table [id=fd7be3c5a0bd46a7868f7d3092b83063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:29.930442 5806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c5e6e3133b841a2ae1d9399e52f13c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:29.931818 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.913854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38461 (local address 127.0.0.1:37108)
0925 15:07:29.923768 (+ 9914us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:29.923786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:29.923880 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:29.925777 (+ 1897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:29.925781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:29.925808 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:29.926044 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.926053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.927569 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.927574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.929344 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.929359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.931102 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.931108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.931282 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.931315 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:07:29.931410 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:07:29.931549 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9821,"thread_start_us":10307,"threads_started":1}
I20260925 15:07:29.931959 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.914136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.22.236.1:38461)
0925 15:07:29.925138 (+ 11002us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:29.925145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:29.925169 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:07:29.925246 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:29.925251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:29.925474 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:29.925666 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:29.926184 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.927409 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.929509 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.930917 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.931064 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.931245 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:29.931464 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 15:07:29.931641 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10860,"thread_start_us":10730,"threads_started":1}
I20260925 15:07:29.936055 5543 tablet_service.cc:1467] Processing CreateTablet for tablet 4c5e6e3133b841a2ae1d9399e52f13c4 (DEFAULT_TABLE table=test-table [id=fd7be3c5a0bd46a7868f7d3092b83063]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:29.937081 5543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c5e6e3133b841a2ae1d9399e52f13c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:29.943109 5887 tablet_bootstrap.cc:492] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a: Bootstrap starting.
I20260925 15:07:29.946671 5888 tablet_bootstrap.cc:492] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b: Bootstrap starting.
I20260925 15:07:29.947347 5887 tablet_bootstrap.cc:654] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:29.948546 5887 log.cc:824] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:29.949402 5888 tablet_bootstrap.cc:654] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:29.950541 5888 log.cc:824] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:29.951076 5890 tablet_bootstrap.cc:492] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6: Bootstrap starting.
I20260925 15:07:29.952216 5887 tablet_bootstrap.cc:492] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a: No bootstrap required, opened a new log
I20260925 15:07:29.952443 5887 ts_tablet_manager.cc:1397] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260925 15:07:29.953893 5890 tablet_bootstrap.cc:654] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:29.954295 5888 tablet_bootstrap.cc:492] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b: No bootstrap required, opened a new log
I20260925 15:07:29.954555 5888 ts_tablet_manager.cc:1397] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 15:07:29.954806 5890 log.cc:824] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:29.957021 5890 tablet_bootstrap.cc:492] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6: No bootstrap required, opened a new log
I20260925 15:07:29.957240 5890 ts_tablet_manager.cc:1397] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260925 15:07:29.962406 5887 raft_consensus.cc:348] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:29.962836 5887 raft_consensus.cc:374] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:29.962945 5887 raft_consensus.cc:729] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d418614b0fbc4b278b47966ab6ad4e3a, State: Initialized, Role: FOLLOWER
I20260925 15:07:29.963548 5887 consensus_queue.cc:260] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:29.964639 5740 heartbeater.cc:503] Master 127.22.236.62:37191 was elected leader, sending a full tablet report...
I20260925 15:07:29.964592 5890 raft_consensus.cc:348] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:29.964936 5890 raft_consensus.cc:374] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:29.964715 5888 raft_consensus.cc:348] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:29.965025 5890 raft_consensus.cc:729] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd21e894bc2d48fd95d5ed25314f52f6, State: Initialized, Role: FOLLOWER
I20260925 15:07:29.965153 5888 raft_consensus.cc:374] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:29.965276 5888 raft_consensus.cc:729] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d853cda79b1a410abd65083195ef4f3b, State: Initialized, Role: FOLLOWER
I20260925 15:07:29.965659 5890 consensus_queue.cc:260] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:29.965857 5888 consensus_queue.cc:260] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:29.966737 5887 ts_tablet_manager.cc:1428] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260925 15:07:29.967613 5871 heartbeater.cc:503] Master 127.22.236.62:37191 was elected leader, sending a full tablet report...
I20260925 15:07:29.968277 5888 ts_tablet_manager.cc:1428] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260925 15:07:29.970961 5608 heartbeater.cc:503] Master 127.22.236.62:37191 was elected leader, sending a full tablet report...
I20260925 15:07:29.972297 5890 ts_tablet_manager.cc:1428] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260925 15:07:30.003436 5895 raft_consensus.cc:482] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:30.003754 5895 raft_consensus.cc:504] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:30.005925 5895 leader_election.cc:288] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413), cd21e894bc2d48fd95d5ed25314f52f6 (127.22.236.1:38461)
W20260925 15:07:30.007323 5609 tablet.cc:2369] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:07:30.012975 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.006268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:39413 (local address 127.22.236.3:50151)
0925 15:07:30.006478 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:30.006497 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:30.006631 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:30.007163 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:30.007168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:30.007195 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:30.007424 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:30.007436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.008868 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.008874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:30.010761 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:30.010780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.012263 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.012269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:30.012457 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:30.012493 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:07:30.012579 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:07:30.012712 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 15:07:30.012980 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.006267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:50151 (local address 127.22.236.2:39413)
0925 15:07:30.006483 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:30.006489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:30.006580 (+ 91us) server_negotiation.cc:407] Connection header received
0925 15:07:30.006807 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:30.006813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:30.006899 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:30.006977 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:30.007682 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.008707 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.010914 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.012108 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.012286 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:30.012360 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:30.012622 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 15:07:30.012748 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 15:07:30.014042 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.006945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:36995 (local address 127.22.236.1:38461)
0925 15:07:30.007153 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:30.007159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:30.007879 (+ 720us) server_negotiation.cc:407] Connection header received
0925 15:07:30.008048 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:30.008053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:30.008127 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:30.008234 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:30.008972 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.010100 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.011899 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.013076 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.013217 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:30.013285 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:30.013668 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 15:07:30.013803 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:07:30.014050 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.006710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38461 (local address 127.22.236.3:36995)
0925 15:07:30.007752 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:30.007767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:30.007915 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:30.008388 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:30.008393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:30.008418 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:30.008781 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:30.008792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.010280 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.010285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:30.011751 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:30.011763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.013363 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.013367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:30.013505 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:30.013535 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:07:30.013628 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:07:30.013743 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":626,"threads_started":1}
I20260925 15:07:30.013973 5695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5e6e3133b841a2ae1d9399e52f13c4" candidate_uuid: "d853cda79b1a410abd65083195ef4f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" is_pre_election: true
I20260925 15:07:30.014436 5695 raft_consensus.cc:2454] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d853cda79b1a410abd65083195ef4f3b in term 0.
I20260925 15:07:30.015082 5761 leader_election.cc:302] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d418614b0fbc4b278b47966ab6ad4e3a, d853cda79b1a410abd65083195ef4f3b; no voters:
I20260925 15:07:30.014961 5563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5e6e3133b841a2ae1d9399e52f13c4" candidate_uuid: "d853cda79b1a410abd65083195ef4f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" is_pre_election: true
I20260925 15:07:30.015470 5895 raft_consensus.cc:2781] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:30.015623 5895 raft_consensus.cc:482] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:30.015556 5563 raft_consensus.cc:2454] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d853cda79b1a410abd65083195ef4f3b in term 0.
I20260925 15:07:30.015730 5895 raft_consensus.cc:3037] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:30.018798 5895 raft_consensus.cc:504] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:30.019461 5895 leader_election.cc:288] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [CANDIDATE]: Term 1 election: Requested vote from peers d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413), cd21e894bc2d48fd95d5ed25314f52f6 (127.22.236.1:38461)
I20260925 15:07:30.019833 5695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5e6e3133b841a2ae1d9399e52f13c4" candidate_uuid: "d853cda79b1a410abd65083195ef4f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d418614b0fbc4b278b47966ab6ad4e3a"
I20260925 15:07:30.019969 5563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5e6e3133b841a2ae1d9399e52f13c4" candidate_uuid: "d853cda79b1a410abd65083195ef4f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd21e894bc2d48fd95d5ed25314f52f6"
I20260925 15:07:30.020025 5695 raft_consensus.cc:3037] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:30.020179 5563 raft_consensus.cc:3037] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:30.022282 5563 raft_consensus.cc:2454] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d853cda79b1a410abd65083195ef4f3b in term 1.
I20260925 15:07:30.022280 5695 raft_consensus.cc:2454] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d853cda79b1a410abd65083195ef4f3b in term 1.
I20260925 15:07:30.022601 5761 leader_election.cc:302] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d418614b0fbc4b278b47966ab6ad4e3a, d853cda79b1a410abd65083195ef4f3b; no voters:
I20260925 15:07:30.022845 5895 raft_consensus.cc:2781] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:30.023249 5895 raft_consensus.cc:686] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [term 1 LEADER]: Becoming Leader. State: Replica: d853cda79b1a410abd65083195ef4f3b, State: Running, Role: LEADER
I20260925 15:07:30.023658 5895 consensus_queue.cc:237] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:30.028251 5422 catalog_manager.cc:5495] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b reported cstate change: term changed from 0 to 1, leader changed from <none> to d853cda79b1a410abd65083195ef4f3b (127.22.236.3). New cstate: current_term: 1 leader_uuid: "d853cda79b1a410abd65083195ef4f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } health_report { overall_health: UNKNOWN } } }
I20260925 15:07:30.043393 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.038622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.22.236.3:37745)
0925 15:07:30.038806 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:30.038812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:30.038828 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:07:30.038904 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:30.038907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:30.038980 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:30.039098 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:30.039464 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.040184 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.041473 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.041883 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.042027 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:30.042453 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:30.042952 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:30.043081 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:30.043150 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 15:07:30.043213 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:07:30.051120 5563 raft_consensus.cc:1261] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Refusing update from remote peer d853cda79b1a410abd65083195ef4f3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:30.051120 5695 raft_consensus.cc:1261] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a [term 1 FOLLOWER]: Refusing update from remote peer d853cda79b1a410abd65083195ef4f3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:30.052033 5897 consensus_queue.cc:1035] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:30.052397 5895 consensus_queue.cc:1035] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:30.075160 5422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:30.075446 5422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:30.078904 5422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Sending DeleteTablet for 3 replicas of tablet 4c5e6e3133b841a2ae1d9399e52f13c4
I20260925 15:07:30.080024 5806 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c5e6e3133b841a2ae1d9399e52f13c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:30 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:38144
I20260925 15:07:30.080304 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 5459
I20260925 15:07:30.080356 5675 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c5e6e3133b841a2ae1d9399e52f13c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:30 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:42738
I20260925 15:07:30.080986 5543 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c5e6e3133b841a2ae1d9399e52f13c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:30 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:37108
I20260925 15:07:30.092708 5908 tablet_replica.cc:330] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b: stopping tablet replica
I20260925 15:07:30.093353 5908 raft_consensus.cc:2229] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:30.094151 5908 raft_consensus.cc:2258] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:30.095052 5909 tablet_replica.cc:330] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a: stopping tablet replica
I20260925 15:07:30.095793 5910 tablet_replica.cc:330] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6: stopping tablet replica
I20260925 15:07:30.096325 5909 raft_consensus.cc:2229] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:30.096925 5909 raft_consensus.cc:2258] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:30.097002 5908 ts_tablet_manager.cc:1905] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:30.096553 5910 raft_consensus.cc:2229] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:30.097873 5910 raft_consensus.cc:2258] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:30.098341 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.081851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.22.236.1:38461)
0925 15:07:30.081996 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:30.082001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:30.082018 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:07:30.082069 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:30.082073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:30.082152 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:30.082248 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:30.083975 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.085174 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.087134 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.087765 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.087890 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:30.091232 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:30.091249 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:30.091268 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:30.091308 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:30.095245 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:30.096099 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:30.096108 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:30.096111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:30.096273 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:30.096652 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:30.096657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:30.096659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:30.097154 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:30.097429 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:30.097764 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 15:07:30.097989 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:07:30.099052 5603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:30.100968 5909 ts_tablet_manager.cc:1905] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:30.104486 5908 ts_tablet_manager.cc:1918] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:30.104655 5908 log.cc:1196] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/4c5e6e3133b841a2ae1d9399e52f13c4
I20260925 15:07:30.105207 5908 ts_tablet_manager.cc:1939] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d853cda79b1a410abd65083195ef4f3b: Deleting consensus metadata
I20260925 15:07:30.107753 5909 ts_tablet_manager.cc:1918] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:30.107991 5909 log.cc:1196] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/4c5e6e3133b841a2ae1d9399e52f13c4
I20260925 15:07:30.108578 5909 ts_tablet_manager.cc:1939] T 4c5e6e3133b841a2ae1d9399e52f13c4 P d418614b0fbc4b278b47966ab6ad4e3a: Deleting consensus metadata
I20260925 15:07:30.106915 5407 catalog_manager.cc:4841] TS d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745): tablet 4c5e6e3133b841a2ae1d9399e52f13c4 (table test-table [id=fd7be3c5a0bd46a7868f7d3092b83063]) successfully deleted
I20260925 15:07:30.110249 5408 catalog_manager.cc:4841] TS d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413): tablet 4c5e6e3133b841a2ae1d9399e52f13c4 (table test-table [id=fd7be3c5a0bd46a7868f7d3092b83063]) successfully deleted
I20260925 15:07:30.198006 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 5459
I20260925 15:07:30.212235 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 5612
W20260925 15:07:30.212273 5408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.236.1:38461 (error 108)
I20260925 15:07:30.214267 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.213729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38461 (local address 127.0.0.1:37130)
0925 15:07:30.213971 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:30.214080 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260925 15:07:30.214521 5408 catalog_manager.cc:4584] TS cd21e894bc2d48fd95d5ed25314f52f6 (127.22.236.1:38461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c5e6e3133b841a2ae1d9399e52f13c4: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38461: connect: Connection refused (error 111)
I20260925 15:07:30.226723 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.213276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42746 (local address 127.22.236.2:39413)
0925 15:07:30.213455 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:30.213461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:30.213481 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:07:30.213620 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:30.213624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:30.213703 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:30.213799 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:30.215270 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.215984 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.217274 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.217668 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.217776 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:30.220330 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:30.220348 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:30.220369 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:30.220416 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:30.223827 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:30.224709 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:30.224720 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:30.224723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:30.224896 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:30.225216 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:30.225221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:30.225225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:30.225687 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:30.225961 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:30.226143 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 15:07:30.226355 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:07:30.227399 5735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:30.268172 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.267631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38461 (local address 127.0.0.1:37138)
0925 15:07:30.267868 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:30.267977 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 15:07:30.269515 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.269001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38461 (local address 127.0.0.1:37146)
0925 15:07:30.269256 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:30.269354 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260925 15:07:30.330397 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.329818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38461 (local address 127.0.0.1:37148)
0925 15:07:30.330111 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:30.330214 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260925 15:07:30.331542 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.331102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38461 (local address 127.0.0.1:37158)
0925 15:07:30.331294 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:30.331369 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 15:07:30.333653 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 5612
I20260925 15:07:30.347596 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 5743
I20260925 15:07:30.363080 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.348562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.22.236.3:37745)
0925 15:07:30.348773 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:30.348779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:30.348800 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:07:30.348871 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:30.348876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:30.348967 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:30.349076 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:30.350348 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.351606 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.352883 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.353576 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.353709 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:30.356276 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:30.356292 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:30.356312 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:30.356361 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:30.359986 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:30.360910 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:30.360921 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:30.360925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:30.361106 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:30.361435 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:30.361439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:30.361442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:30.361897 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:30.362197 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:30.362421 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 15:07:30.362671 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:07:30.363765 5866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:30.412117 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.411582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38461 (local address 127.0.0.1:37170)
0925 15:07:30.411813 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:30.411929 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 15:07:30.413239 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.412764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38461 (local address 127.0.0.1:37186)
0925 15:07:30.412985 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:30.413080 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260925 15:07:30.482497 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 5743
I20260925 15:07:30.497285 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 5389
I20260925 15:07:30.498297 5452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:30.608363 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.607662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38461 (local address 127.0.0.1:37202)
0925 15:07:30.608037 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:30.608191 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260925 15:07:30.608374 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 5389
I20260925 15:07:30.625213 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:37191
--webserver_interface=127.22.236.62
--webserver_port=45329
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:37191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:30.933699 5916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:30.933995 5916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:30.934134 5916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:30.941280 5916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:30.941401 5916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:30.941483 5916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:30.941567 5916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:30.941628 5916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:30.951267 5916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:37191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:37191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=45329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:30.952555 5916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:30.953830 5916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:30.964941 5921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:30.964984 5922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:30.965018 5924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:30.965826 5916 server_base.cc:1034] running on GCE node
I20260925 15:07:30.966590 5916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:30.967923 5916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:30.969138 5916 hybrid_clock.cc:648] HybridClock initialized: now 1790348850969073 us; error 49 us; skew 500 ppm
I20260925 15:07:30.969462 5916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:30 dist-test-slave-nsrc krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348850, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:30.976158 5916 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:07:30.978070 5916 webserver.cc:460] Webserver started at http://127.22.236.62:45329/ using document root <none> and password file <none>
I20260925 15:07:30.978740 5916 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:30.978852 5916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:30.984297 5916 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:07:30.987021 5931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:30.988381 5916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:30.988525 5916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01"
format_stamp: "Formatted at 2026-09-25 15:07:25 on dist-test-slave-nsrc"
server_key: "f326277dca769a123bfc3fc4b642c232"
server_key_iv: "b5dc5565645e1b6c38988284d9b4502d"
server_key_version: "encryptionkey@0"
I20260925 15:07:30.989444 5916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:31.001571 5916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:31.005673 5916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:31.005913 5916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:31.022771 5916 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:37191
I20260925 15:07:31.022789 5982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:37191 every 8 connection(s)
I20260925 15:07:31.024209 5916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:31.030896 5983 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:07:31.034067 5983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Bootstrap starting.
I20260925 15:07:31.034502 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 5916
I20260925 15:07:31.035252 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:38461
--local_ip_for_outbound_sockets=127.22.236.1
--tserver_master_addrs=127.22.236.62:37191
--webserver_port=35875
--webserver_interface=127.22.236.1
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:07:31.053983 5983 log.cc:824] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:31.069917 5983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:07:31.070407 5983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01: Bootstrap complete.
I20260925 15:07:31.079047 5983 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } }
I20260925 15:07:31.079875 5983 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5bb7cbc833d49b3bdb37dbc7534bf01, State: Initialized, Role: FOLLOWER
I20260925 15:07:31.080507 5983 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } }
I20260925 15:07:31.080725 5983 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:31.080835 5983 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:31.080986 5983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:07:31.085065 5983 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } }
I20260925 15:07:31.085631 5983 leader_election.cc:302] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5bb7cbc833d49b3bdb37dbc7534bf01; no voters:
I20260925 15:07:31.086381 5983 leader_election.cc:288] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:07:31.086588 5987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:07:31.089600 5987 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [term 3 LEADER]: Becoming Leader. State: Replica: b5bb7cbc833d49b3bdb37dbc7534bf01, State: Running, Role: LEADER
I20260925 15:07:31.090467 5987 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } }
I20260925 15:07:31.094393 5988 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5bb7cbc833d49b3bdb37dbc7534bf01. Latest consensus state: current_term: 3 leader_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } } }
I20260925 15:07:31.094777 5988 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:31.095234 5983 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:31.095257 5989 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bb7cbc833d49b3bdb37dbc7534bf01" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37191 } } }
I20260925 15:07:31.095768 5989 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5bb7cbc833d49b3bdb37dbc7534bf01 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:31.096906 5991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:31.118664 5991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:31.120062 5991 catalog_manager.cc:1259] Loaded cluster ID: 865919df54e544b6b9d139d65754e1fe
I20260925 15:07:31.120189 5991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 15:07:31.122007 5991 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbefeb339ad google::LogMessage::Fail() at ??:0
@ 0x7fbefeb3688a google::LogMessage::SendToLog() at ??:0
@ 0x7fbefeb334de google::LogMessage::Flush() at ??:0
@ 0x7fbefeb34a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbf14566512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fbf14563976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbf145eb610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fbf145eb439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbf0177f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbf0178291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fbf01782799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbf0175deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbf04a376db start_thread at ??:0
@ 0x7fbefd9cd61f clone at ??:0
W20260925 15:07:31.406368 5985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:31.406639 5985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:31.406755 5985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:31.413995 5985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:31.414108 5985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:31.414254 5985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:07:31.423863 5985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:38461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=35875
--tserver_master_addrs=127.22.236.62:37191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:31.425207 5985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:31.426519 5985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:31.439348 6023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:31.439288 6024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:31.439481 6026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:31.440110 5985 server_base.cc:1034] running on GCE node
I20260925 15:07:31.440725 5985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:31.441597 5985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:31.442791 5985 hybrid_clock.cc:648] HybridClock initialized: now 1790348851442739 us; error 40 us; skew 500 ppm
I20260925 15:07:31.443091 5985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:31 dist-test-slave-nsrc krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348851, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:31.446113 5985 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:07:31.447386 5985 webserver.cc:460] Webserver started at http://127.22.236.1:35875/ using document root <none> and password file <none>
I20260925 15:07:31.447999 5985 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:31.448101 5985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:31.453243 5985 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:07:31.455868 6033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:31.457149 5985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:07:31.457293 5985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "cd21e894bc2d48fd95d5ed25314f52f6"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-nsrc"
server_key: "7e515b424c8fa984df7103c45aec1bf1"
server_key_iv: "183d50e4738a30f8fdf9463bd626c304"
server_key_version: "encryptionkey@0"
I20260925 15:07:31.458271 5985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:31.470844 5985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:31.474826 5985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:31.475077 5985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:31.475965 5985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:31.477655 6040 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 15:07:31.482856 5985 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:07:31.482990 5985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260925 15:07:31.483107 5985 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:07:31.486306 5985 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:07:31.486382 5985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260925 15:07:31.486702 6040 tablet_bootstrap.cc:492] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6: Bootstrap starting.
I20260925 15:07:31.494958 6040 log.cc:824] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:31.509253 6040 tablet_bootstrap.cc:492] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:07:31.510178 6040 tablet_bootstrap.cc:492] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6: Bootstrap complete.
I20260925 15:07:31.510758 6040 ts_tablet_manager.cc:1397] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.010s
I20260925 15:07:31.520794 5985 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:38461
I20260925 15:07:31.520848 6147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:38461 every 8 connection(s)
I20260925 15:07:31.520107 6040 raft_consensus.cc:348] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:31.521261 6040 raft_consensus.cc:729] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd21e894bc2d48fd95d5ed25314f52f6, State: Initialized, Role: FOLLOWER
I20260925 15:07:31.521904 6040 consensus_queue.cc:260] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:31.522142 5985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:31.523691 6040 ts_tablet_manager.cc:1428] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260925 15:07:31.526665 6151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:31.524904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:43011)
0925 15:07:31.525608 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:31.525832 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":289,"threads_started":1}
W20260925 15:07:31.527283 6149 heartbeater.cc:644] Failed to heartbeat to 127.22.236.62:37191 (0 consecutive failures): Network error: Failed to ping master at 127.22.236.62:37191: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
I20260925 15:07:31.528059 6151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:31.527746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:33903)
0925 15:07:31.527868 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:31.527963 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:07:31.528908 6151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:31.528581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:40047)
0925 15:07:31.528721 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:31.528789 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:07:31.529134 6149 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:07:31.531611 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 5985
I20260925 15:07:31.532176 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:39413
--local_ip_for_outbound_sockets=127.22.236.2
--tserver_master_addrs=127.22.236.62:37191
--webserver_port=33073
--webserver_interface=127.22.236.2
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:31.835155 6152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:31.835421 6152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:31.835546 6152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:31.842701 6152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:31.842803 6152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:31.842947 6152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:07:31.852460 6152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:39413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=33073
--tserver_master_addrs=127.22.236.62:37191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:31.853668 6152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:31.854907 6152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:31.867460 6158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:31.867487 6160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:31.867427 6157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:31.868135 6152 server_base.cc:1034] running on GCE node
I20260925 15:07:31.868723 6152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:31.869668 6152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:31.870859 6152 hybrid_clock.cc:648] HybridClock initialized: now 1790348851870801 us; error 49 us; skew 500 ppm
I20260925 15:07:31.871176 6152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:31 dist-test-slave-nsrc krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348851, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:31.874236 6152 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:07:31.875579 6152 webserver.cc:460] Webserver started at http://127.22.236.2:33073/ using document root <none> and password file <none>
I20260925 15:07:31.876247 6152 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:31.876348 6152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:31.881440 6152 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:07:31.884032 6167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:31.885329 6152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:31.885478 6152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "d418614b0fbc4b278b47966ab6ad4e3a"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-nsrc"
server_key: "1635bed02e64f03fda8e4008c12f9f92"
server_key_iv: "a7bf98007b6ca62a2e83efb592d34371"
server_key_version: "encryptionkey@0"
I20260925 15:07:31.886386 6152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:31.906188 6152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:31.910271 6152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:31.910516 6152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:31.911468 6152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:31.912758 6152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:31.912858 6152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:31.912976 6152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:31.913043 6152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:31.942245 6152 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:39413
I20260925 15:07:31.942317 6279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:39413 every 8 connection(s)
I20260925 15:07:31.943653 6152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:31.948195 6282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:31.946471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:40863)
0925 15:07:31.947146 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:31.947352 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":250,"threads_started":1}
W20260925 15:07:31.948808 6280 heartbeater.cc:644] Failed to heartbeat to 127.22.236.62:37191 (0 consecutive failures): Network error: Failed to ping master at 127.22.236.62:37191: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
I20260925 15:07:31.949496 6282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:31.949222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:34515)
0925 15:07:31.949337 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:31.949400 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:07:31.950198 6282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:31.949939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:42939)
0925 15:07:31.950038 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:31.950103 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 15:07:31.950382 6280 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:07:31.951596 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 6152
I20260925 15:07:31.952134 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:37745
--local_ip_for_outbound_sockets=127.22.236.3
--tserver_master_addrs=127.22.236.62:37191
--webserver_port=33903
--webserver_interface=127.22.236.3
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:32.261094 6283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:32.261363 6283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:32.261510 6283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:32.268784 6283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:32.268899 6283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:32.269063 6283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:07:32.278704 6283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:37745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=33903
--tserver_master_addrs=127.22.236.62:37191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:32.279906 6283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:32.281152 6283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:32.293457 6288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:32.293522 6289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:32.293561 6291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:32.294339 6283 server_base.cc:1034] running on GCE node
I20260925 15:07:32.294906 6283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:32.295734 6283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:32.296926 6283 hybrid_clock.cc:648] HybridClock initialized: now 1790348852296887 us; error 28 us; skew 500 ppm
I20260925 15:07:32.297222 6283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:32 dist-test-slave-nsrc krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348852, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:32.300189 6283 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:07:32.301527 6283 webserver.cc:460] Webserver started at http://127.22.236.3:33903/ using document root <none> and password file <none>
I20260925 15:07:32.302130 6283 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:32.302225 6283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:32.307328 6283 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 15:07:32.309847 6298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:32.311096 6283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:07:32.311240 6283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "d853cda79b1a410abd65083195ef4f3b"
format_stamp: "Formatted at 2026-09-25 15:07:27 on dist-test-slave-nsrc"
server_key: "a32137ee11139b0361f7a0c49e479141"
server_key_iv: "4caed2259640fa3a2eed5bd48fbc1020"
server_key_version: "encryptionkey@0"
I20260925 15:07:32.312183 6283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:32.330708 6283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:32.334633 6283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:32.334887 6283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:32.335783 6283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:32.337055 6283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:32.337145 6283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:32.337251 6283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:32.337316 6283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:32.366466 6283 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:37745
I20260925 15:07:32.366523 6410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:37745 every 8 connection(s)
I20260925 15:07:32.367810 6283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:32.370999 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 6283
I20260925 15:07:32.372575 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:32.370586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:53747)
0925 15:07:32.371367 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:32.371614 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":309,"threads_started":1}
W20260925 15:07:32.373199 6411 heartbeater.cc:644] Failed to heartbeat to 127.22.236.62:37191 (0 consecutive failures): Network error: Failed to ping master at 127.22.236.62:37191: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
I20260925 15:07:32.373920 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:32.373611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:41807)
0925 15:07:32.373752 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:32.373813 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:07:32.374630 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:32.374335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:58525)
0925 15:07:32.374463 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:32.374532 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:07:32.374826 6411 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:07:32.530526 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:32.529647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:40923)
0925 15:07:32.530098 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:32.530323 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":225,"threads_started":1}
I20260925 15:07:32.775053 6416 raft_consensus.cc:482] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:32.775333 6416 raft_consensus.cc:504] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:32.776986 6416 leader_election.cc:288] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413), d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745)
Sep 25 15:07:32 dist-test-slave-nsrc krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348851, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
Sep 25 15:07:32 dist-test-slave-nsrc krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348851, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:07:32.798002 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:32.777323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:39413 (local address 127.22.236.1:55931)
0925 15:07:32.779167 (+ 1844us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:32.779181 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:32.779300 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:32.781916 (+ 2616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:32.781926 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:32.782668 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:32.783412 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:32.783436 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:32.784965 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:32.784969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:32.786340 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:32.786352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:32.787299 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:32.787306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:32.787513 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:32.788265 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:32.788287 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:32.790744 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:32.794310 (+ 3566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:32.794315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:32.794319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:32.794777 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:32.795172 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:32.795176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:32.795179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:32.795376 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:32.796284 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:32.796292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:32.796743 (+ 451us) client_negotiation.cc:769] Sending connection context
0925 15:07:32.797047 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 15:07:32.797551 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1707,"mutex_wait_us":176,"spinlock_wait_cycles":23680,"thread_start_us":1544,"threads_started":1}
I20260925 15:07:32.798107 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:32.777461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:55931 (local address 127.22.236.2:39413)
0925 15:07:32.778909 (+ 1448us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:32.778926 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:32.779283 (+ 357us) server_negotiation.cc:407] Connection header received
0925 15:07:32.780999 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:32.781011 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:32.781449 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:32.781758 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:32.783599 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:32.784778 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:32.786467 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:32.787068 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:32.787232 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:32.791042 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:32.791051 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:32.791065 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:32.791099 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:32.794191 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:32.794894 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:32.794902 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:32.794907 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:32.795064 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:32.795475 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:32.795480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:32.795484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:32.796104 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:32.796309 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:32.797292 (+ 983us) server_negotiation.cc:299] Negotiation successful
0925 15:07:32.797691 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1240,"thread_start_us":1111,"threads_started":1}
I20260925 15:07:32.798101 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:32.777638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:35199 (local address 127.22.236.3:37745)
0925 15:07:32.779401 (+ 1763us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:32.779411 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:32.779444 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:07:32.781066 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:32.781077 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:32.781449 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:32.781758 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:32.783600 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:32.784788 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:32.786473 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:32.787063 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:32.787223 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:32.791320 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:32.791328 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:32.791340 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:32.791375 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:32.794109 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:32.794890 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:32.794897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:32.794902 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:32.795052 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:32.795473 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:32.795476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:32.795478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:32.796115 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:32.796311 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:32.797292 (+ 981us) server_negotiation.cc:299] Negotiation successful
0925 15:07:32.797696 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1620,"thread_start_us":1474,"threads_started":1}
I20260925 15:07:32.798038 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:32.777267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:37745 (local address 127.22.236.1:35199)
0925 15:07:32.777528 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:32.777572 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:32.778150 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:32.781917 (+ 3767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:32.781926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:32.782653 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:32.783413 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:32.783436 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:32.784960 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:32.784964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:32.786338 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:32.786350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:32.787265 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:32.787275 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:32.787517 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:32.788240 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:32.788267 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:32.791088 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:32.794241 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:32.794251 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:32.794266 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:32.794765 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:32.795165 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:32.795170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:32.795174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:32.795370 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:32.796382 (+ 1012us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:32.796387 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:32.796743 (+ 356us) client_negotiation.cc:769] Sending connection context
0925 15:07:32.797047 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 15:07:32.797685 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":428,"spinlock_wait_cycles":27392}
W20260925 15:07:32.801501 6035 leader_election.cc:341] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745): Not found: Tablet not found: 4c5e6e3133b841a2ae1d9399e52f13c4
W20260925 15:07:32.801752 6036 leader_election.cc:341] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413): Not found: Tablet not found: 4c5e6e3133b841a2ae1d9399e52f13c4
I20260925 15:07:32.801905 6036 leader_election.cc:302] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd21e894bc2d48fd95d5ed25314f52f6; no voters: d418614b0fbc4b278b47966ab6ad4e3a, d853cda79b1a410abd65083195ef4f3b
I20260925 15:07:32.802258 6416 raft_consensus.cc:2726] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:07:32.952118 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:32.951160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:36003)
0925 15:07:32.951645 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:32.951812 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":233,"threads_started":1}
I20260925 15:07:33.375964 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:33.375296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:51493)
0925 15:07:33.375674 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:33.375782 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":179,"threads_started":1}
I20260925 15:07:33.531981 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:33.531341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:55711)
0925 15:07:33.531742 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:33.531821 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":211,"threads_started":1}
I20260925 15:07:33.953675 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:33.952994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:35535)
0925 15:07:33.953443 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:33.953521 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":256,"threads_started":1}
I20260925 15:07:34.377473 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:34.376795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:42915)
0925 15:07:34.377219 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:34.377300 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":217,"threads_started":1}
I20260925 15:07:34.533479 6425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:34.532800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:33387)
0925 15:07:34.533230 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:34.533306 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":219,"threads_started":1}
I20260925 15:07:34.534068 6426 raft_consensus.cc:482] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:34.534240 6426 raft_consensus.cc:504] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:34.534811 6426 leader_election.cc:288] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413), d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745)
W20260925 15:07:34.535641 6035 leader_election.cc:341] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745): Not found: Tablet not found: 4c5e6e3133b841a2ae1d9399e52f13c4
W20260925 15:07:34.535810 6036 leader_election.cc:341] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413): Not found: Tablet not found: 4c5e6e3133b841a2ae1d9399e52f13c4
I20260925 15:07:34.535905 6036 leader_election.cc:302] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd21e894bc2d48fd95d5ed25314f52f6; no voters: d418614b0fbc4b278b47966ab6ad4e3a, d853cda79b1a410abd65083195ef4f3b
I20260925 15:07:34.536108 6426 raft_consensus.cc:2726] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:07:34.955158 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:34.954520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:39227)
0925 15:07:34.954928 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:34.955010 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":197,"threads_started":1}
I20260925 15:07:35.378985 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:35.378316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:60899)
0925 15:07:35.378746 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:35.378831 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":228,"threads_started":1}
I20260925 15:07:35.534997 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:35.534289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:39007)
0925 15:07:35.534731 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:35.534816 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":231,"threads_started":1}
I20260925 15:07:35.956796 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:35.956092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:38771)
0925 15:07:35.956542 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:35.956623 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":234,"threads_started":1}
I20260925 15:07:36.380612 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:36.379842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:56911)
0925 15:07:36.380317 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:36.380423 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":228,"threads_started":1}
I20260925 15:07:36.536773 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:36.535979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:54393)
0925 15:07:36.536502 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:36.536601 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":248,"threads_started":1}
I20260925 15:07:36.958384 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:36.957665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:46967)
0925 15:07:36.958129 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:36.958223 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":242,"threads_started":1}
I20260925 15:07:37.070405 6434 raft_consensus.cc:482] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:37.070598 6434 raft_consensus.cc:504] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:37.071331 6434 leader_election.cc:288] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413), d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745)
W20260925 15:07:37.072222 6036 leader_election.cc:341] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413): Not found: Tablet not found: 4c5e6e3133b841a2ae1d9399e52f13c4
W20260925 15:07:37.072418 6035 leader_election.cc:341] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745): Not found: Tablet not found: 4c5e6e3133b841a2ae1d9399e52f13c4
I20260925 15:07:37.072559 6035 leader_election.cc:302] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd21e894bc2d48fd95d5ed25314f52f6; no voters: d418614b0fbc4b278b47966ab6ad4e3a, d853cda79b1a410abd65083195ef4f3b
I20260925 15:07:37.072800 6434 raft_consensus.cc:2726] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:07:37.382117 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:37.381469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:40467)
0925 15:07:37.381870 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:37.381953 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":208,"threads_started":1}
I20260925 15:07:37.538439 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:37.537637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:51279)
0925 15:07:37.538149 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:37.538250 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":277,"threads_started":1}
I20260925 15:07:37.959877 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:37.959212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:60087)
0925 15:07:37.959627 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:37.959707 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":218,"threads_started":1}
I20260925 15:07:38.383786 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:38.383036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:33525)
0925 15:07:38.383500 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:38.383601 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":211,"threads_started":1}
I20260925 15:07:38.540030 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:38.539257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:36137)
0925 15:07:38.539710 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:38.539803 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":227,"threads_started":1}
I20260925 15:07:38.961534 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:38.960724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:56943)
0925 15:07:38.961254 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:38.961346 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":229,"threads_started":1}
I20260925 15:07:39.385599 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.384783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:38855)
0925 15:07:39.385299 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.385384 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":209,"threads_started":1}
I20260925 15:07:39.541571 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.540867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:40961)
0925 15:07:39.541297 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.541402 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":210,"threads_started":1}
I20260925 15:07:39.800258 6443 raft_consensus.cc:482] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:39.800431 6443 raft_consensus.cc:504] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:39.801050 6443 leader_election.cc:288] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413), d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745)
W20260925 15:07:39.801887 6036 leader_election.cc:341] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413): Not found: Tablet not found: 4c5e6e3133b841a2ae1d9399e52f13c4
W20260925 15:07:39.802063 6035 leader_election.cc:341] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745): Not found: Tablet not found: 4c5e6e3133b841a2ae1d9399e52f13c4
I20260925 15:07:39.802156 6035 leader_election.cc:302] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd21e894bc2d48fd95d5ed25314f52f6; no voters: d418614b0fbc4b278b47966ab6ad4e3a, d853cda79b1a410abd65083195ef4f3b
I20260925 15:07:39.802348 6443 raft_consensus.cc:2726] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:07:39.963120 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.962401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:44063)
0925 15:07:39.962855 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.962936 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":256,"threads_started":1}
I20260925 15:07:40.387064 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.386424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:45637)
0925 15:07:40.386819 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:40.386900 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":203,"threads_started":1}
I20260925 15:07:40.543073 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.542356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:53461)
0925 15:07:40.542761 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:40.542843 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":210,"threads_started":1}
I20260925 15:07:40.964699 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.963950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:36409)
0925 15:07:40.964466 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:40.964550 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":272,"threads_started":1}
I20260925 15:07:41.388597 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.387888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:44659)
0925 15:07:41.388357 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:41.388438 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":237,"threads_started":1}
I20260925 15:07:41.544679 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.543945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:45763)
0925 15:07:41.544427 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:41.544516 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":271,"threads_started":1}
I20260925 15:07:41.915423 6450 raft_consensus.cc:482] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:41.915627 6450 raft_consensus.cc:504] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:41.916404 6450 leader_election.cc:288] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413), d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745)
W20260925 15:07:41.917328 6036 leader_election.cc:341] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413): Not found: Tablet not found: 4c5e6e3133b841a2ae1d9399e52f13c4
W20260925 15:07:41.917522 6035 leader_election.cc:341] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745): Not found: Tablet not found: 4c5e6e3133b841a2ae1d9399e52f13c4
I20260925 15:07:41.917644 6035 leader_election.cc:302] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd21e894bc2d48fd95d5ed25314f52f6; no voters: d418614b0fbc4b278b47966ab6ad4e3a, d853cda79b1a410abd65083195ef4f3b
I20260925 15:07:41.917860 6450 raft_consensus.cc:2726] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:07:41.966202 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.965546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:44359)
0925 15:07:41.965936 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:41.966029 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":207,"threads_started":1}
I20260925 15:07:42.390184 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:42.389516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:59301)
0925 15:07:42.389919 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:42.390003 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":203,"threads_started":1}
I20260925 15:07:42.546170 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:42.545524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:57249)
0925 15:07:42.545914 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:42.546000 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":204,"threads_started":1}
I20260925 15:07:42.967798 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:42.967084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:47123)
0925 15:07:42.967550 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:42.967636 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":217,"threads_started":1}
I20260925 15:07:43.391877 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.391162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:51641)
0925 15:07:43.391633 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.391714 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":223,"threads_started":1}
I20260925 15:07:43.547659 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.547014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:42469)
0925 15:07:43.547416 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.547494 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":215,"threads_started":1}
I20260925 15:07:43.969326 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.968657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:39371)
0925 15:07:43.969075 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.969157 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":210,"threads_started":1}
I20260925 15:07:44.342494 6458 raft_consensus.cc:482] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:44.342662 6458 raft_consensus.cc:504] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d418614b0fbc4b278b47966ab6ad4e3a" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 39413 } } peers { permanent_uuid: "d853cda79b1a410abd65083195ef4f3b" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 37745 } } peers { permanent_uuid: "cd21e894bc2d48fd95d5ed25314f52f6" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38461 } }
I20260925 15:07:44.343217 6458 leader_election.cc:288] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413), d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745)
W20260925 15:07:44.344079 6035 leader_election.cc:341] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d853cda79b1a410abd65083195ef4f3b (127.22.236.3:37745): Not found: Tablet not found: 4c5e6e3133b841a2ae1d9399e52f13c4
W20260925 15:07:44.344261 6036 leader_election.cc:341] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d418614b0fbc4b278b47966ab6ad4e3a (127.22.236.2:39413): Not found: Tablet not found: 4c5e6e3133b841a2ae1d9399e52f13c4
I20260925 15:07:44.344394 6036 leader_election.cc:302] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd21e894bc2d48fd95d5ed25314f52f6; no voters: d418614b0fbc4b278b47966ab6ad4e3a, d853cda79b1a410abd65083195ef4f3b
I20260925 15:07:44.344583 6458 raft_consensus.cc:2726] T 4c5e6e3133b841a2ae1d9399e52f13c4 P cd21e894bc2d48fd95d5ed25314f52f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:07:44.393386 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.392657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:48923)
0925 15:07:44.393137 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:44.393220 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":247,"threads_started":1}
I20260925 15:07:44.549345 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.548527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:42763)
0925 15:07:44.549052 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:44.549160 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":215,"threads_started":1}
I20260925 15:07:44.970849 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.970185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:40499)
0925 15:07:44.970595 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:44.970691 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":213,"threads_started":1}
I20260925 15:07:45.395155 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:45.394275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:44157)
0925 15:07:45.394789 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:45.394940 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":254,"threads_started":1}
I20260925 15:07:45.550845 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:45.550177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:34391)
0925 15:07:45.550579 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:45.550661 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":202,"threads_started":1}
I20260925 15:07:45.972554 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:45.971781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:43651)
0925 15:07:45.972252 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:45.972349 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":219,"threads_started":1}
I20260925 15:07:46.396832 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:46.396066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:36017)
0925 15:07:46.396550 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:46.396642 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":241,"threads_started":1}
I20260925 15:07:46.552711 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:46.551913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.1:36003)
0925 15:07:46.552451 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:46.552544 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":284,"threads_started":1}
I20260925 15:07:46.974184 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:46.973510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.2:55187)
0925 15:07:46.973928 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:46.974021 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":215,"threads_started":1}
I20260925 15:07:47.374977 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 5985
I20260925 15:07:47.389611 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:47.375983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.22.236.1:38461)
0925 15:07:47.376579 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:47.376589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:47.376619 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:07:47.376699 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:47.376708 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:47.376991 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:47.377163 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:47.378791 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.379769 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.381033 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.381849 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.381947 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:47.384522 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:47.384533 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:47.384547 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:47.384583 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:47.387017 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:47.387628 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:47.387635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:47.387637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:47.387761 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:47.388181 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:47.388186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:47.388189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:47.388550 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:47.388778 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:47.389050 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 15:07:47.389244 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":307,"threads_started":1}
I20260925 15:07:47.390810 6143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:47.398712 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:47.397980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37191 (local address 127.22.236.3:39695)
0925 15:07:47.398446 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:47.398538 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":228,"threads_started":1}
I20260925 15:07:47.476680 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 5985
I20260925 15:07:47.490100 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 6152
I20260925 15:07:47.503393 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:47.490846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.22.236.2:39413)
0925 15:07:47.491371 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:47.491375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:47.491391 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:07:47.491444 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:47.491447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:47.491507 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:47.491604 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:47.493030 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.493778 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.494941 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.495358 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.495441 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:47.497925 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:47.497931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:47.497934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:47.497964 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:47.500358 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:47.500957 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:47.500960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:47.500963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:47.501108 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:47.502184 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:47.502187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:47.502189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:47.502449 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:47.502643 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:47.502868 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 15:07:47.503051 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":327,"threads_started":1}
I20260925 15:07:47.504043 6275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:47.584905 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 6152
I20260925 15:07:47.598670 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 6283
I20260925 15:07:47.611814 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:47.599485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.22.236.3:37745)
0925 15:07:47.600025 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:47.600030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:47.600045 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:07:47.600095 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:47.600099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:47.600200 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:47.600284 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:47.601562 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.602316 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.603540 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.603941 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.604027 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:47.607127 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:47.607132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:47.607136 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:47.607167 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:47.609711 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:47.610296 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:47.610300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:47.610303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:47.610431 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:47.610710 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:47.610712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:47.610714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:47.610943 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:47.611139 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:47.611324 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 15:07:47.611468 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":289,"threads_started":1}
I20260925 15:07:47.612502 6406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:47.693060 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 6283
2026-09-25T15:07:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23588 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:47 dist-test-slave-nsrc krb5kdc[6477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:47 dist-test-slave-nsrc krb5kdc[6477](info): set up 2 sockets
Sep 25 15:07:47 dist-test-slave-nsrc krb5kdc[6477](info): commencing operation
krb5kdc: starting...
W20260925 15:07:48.700614 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.008s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:07:48 dist-test-slave-nsrc krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:07:48.912276 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:43825
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:43825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:49.221329 6490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:49.221640 6490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:49.221761 6490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:49.229079 6490 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:07:49.229184 6490 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:07:49.229249 6490 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:07:49.229312 6490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:49.229385 6490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:49.229478 6490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:49.229539 6490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:49.229611 6490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:49.239414 6490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:43825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:43825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:49.240693 6490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:49.241966 6490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:49.253454 6498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:49.253521 6496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:49.253535 6495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:49.253741 6490 server_base.cc:1034] running on GCE node
I20260925 15:07:49.254670 6490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:49.256000 6490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:49.257207 6490 hybrid_clock.cc:648] HybridClock initialized: now 1790348869257141 us; error 39 us; skew 500 ppm
I20260925 15:07:49.257501 6490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:49 dist-test-slave-nsrc krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348869, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:49.260413 6490 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:07:49.261781 6490 webserver.cc:460] Webserver started at http://127.22.236.62:43865/ using document root <none> and password file <none>
I20260925 15:07:49.262418 6490 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:49.262512 6490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:49.262794 6490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:49.265647 6490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "90ec8d211ecc4d89852cc61549df1b82"
format_stamp: "Formatted at 2026-09-25 15:07:49 on dist-test-slave-nsrc"
server_key: "615a62a6a6239d9b24603a458b33542c"
server_key_iv: "e78f100c9dbd5245f2098c59f1a6d95a"
server_key_version: "encryptionkey@0"
I20260925 15:07:49.266371 6490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "90ec8d211ecc4d89852cc61549df1b82"
format_stamp: "Formatted at 2026-09-25 15:07:49 on dist-test-slave-nsrc"
server_key: "615a62a6a6239d9b24603a458b33542c"
server_key_iv: "e78f100c9dbd5245f2098c59f1a6d95a"
server_key_version: "encryptionkey@0"
I20260925 15:07:49.271526 6490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:07:49.274780 6505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:49.276036 6490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:07:49.276218 6490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "90ec8d211ecc4d89852cc61549df1b82"
format_stamp: "Formatted at 2026-09-25 15:07:49 on dist-test-slave-nsrc"
server_key: "615a62a6a6239d9b24603a458b33542c"
server_key_iv: "e78f100c9dbd5245f2098c59f1a6d95a"
server_key_version: "encryptionkey@0"
I20260925 15:07:49.276427 6490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:49.290820 6490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:49.294660 6490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:49.294905 6490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:49.311408 6490 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:43825
I20260925 15:07:49.311431 6556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:43825 every 8 connection(s)
I20260925 15:07:49.312696 6490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:49.316217 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:49.320506 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 6490
I20260925 15:07:49.320673 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:07:49.321290 23472 external_mini_cluster.cc:1428] Setting key 4b70488c8c09b7b10e4a106fa1197e06
I20260925 15:07:49.326395 6557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Bootstrap starting.
I20260925 15:07:49.329694 6557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:49.330873 6557 log.cc:824] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:49.334452 6557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: No bootstrap required, opened a new log
Sep 25 15:07:49 dist-test-slave-nsrc krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:49.343840 6557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } }
I20260925 15:07:49.344241 6557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:49.344344 6557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90ec8d211ecc4d89852cc61549df1b82, State: Initialized, Role: FOLLOWER
I20260925 15:07:49.344944 6557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } }
I20260925 15:07:49.345093 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.323063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.22.236.62:43825)
0925 15:07:49.323815 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.323826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.323869 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:07:49.325559 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.325582 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.325934 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.326241 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:49.327945 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.329246 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.331175 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.332235 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.332473 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.337567 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:49.337582 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:49.337604 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:49.337654 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:49.341075 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:49.341867 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:49.341873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:49.341879 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:49.342066 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:49.342436 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:49.342438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:49.342440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:49.342951 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:49.343162 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.343588 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.343910 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":340,"threads_started":1}
I20260925 15:07:49.345156 6557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:49.345275 6557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:49.345460 6557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:49.347653 6557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } }
I20260925 15:07:49.348088 6557 leader_election.cc:302] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90ec8d211ecc4d89852cc61549df1b82; no voters:
I20260925 15:07:49.348645 6557 leader_election.cc:288] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:49.348763 6562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:49.349346 6562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 1 LEADER]: Becoming Leader. State: Replica: 90ec8d211ecc4d89852cc61549df1b82, State: Running, Role: LEADER
I20260925 15:07:49.349824 6562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } }
I20260925 15:07:49.350471 6557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:49.352383 6563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90ec8d211ecc4d89852cc61549df1b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } } }
I20260925 15:07:49.352720 6563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:49.352994 6564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90ec8d211ecc4d89852cc61549df1b82. Latest consensus state: current_term: 1 leader_uuid: "90ec8d211ecc4d89852cc61549df1b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } } }
I20260925 15:07:49.353358 6564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:49.356153 6568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:49.360836 6568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:49.369239 6568 catalog_manager.cc:1347] Generated new cluster ID: bccb7940e50d4243b5fdc71dabd0aebc
I20260925 15:07:49.369350 6568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:49.385834 6568 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:49.387714 6568 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:49.404744 6568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Generated new TSK 0
I20260925 15:07:49.405844 6568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:07:49.458961 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43825
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:49.769377 6583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:49.769634 6583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:49.769724 6583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:49.776839 6583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:49.776944 6583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:49.777056 6583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:07:49.786638 6583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:49.787849 6583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:49.789110 6583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:49.801133 6588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:49.801177 6591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:49.801199 6589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:49.801888 6583 server_base.cc:1034] running on GCE node
I20260925 15:07:49.802446 6583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:49.803304 6583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:49.804502 6583 hybrid_clock.cc:648] HybridClock initialized: now 1790348869804441 us; error 44 us; skew 500 ppm
I20260925 15:07:49.804781 6583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:49 dist-test-slave-nsrc krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348869, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:49.807683 6583 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:07:49.809048 6583 webserver.cc:460] Webserver started at http://127.22.236.1:44327/ using document root <none> and password file <none>
I20260925 15:07:49.809692 6583 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:49.809801 6583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:49.810071 6583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:49.812870 6583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "e91045ac4eda4e99bd38f2aa6df3031c"
format_stamp: "Formatted at 2026-09-25 15:07:49 on dist-test-slave-nsrc"
server_key: "3a40186eb19e5be40cdf0b1ff5f710ee"
server_key_iv: "77ef420232451279ff43ff30b563baf4"
server_key_version: "encryptionkey@0"
I20260925 15:07:49.813575 6583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "e91045ac4eda4e99bd38f2aa6df3031c"
format_stamp: "Formatted at 2026-09-25 15:07:49 on dist-test-slave-nsrc"
server_key: "3a40186eb19e5be40cdf0b1ff5f710ee"
server_key_iv: "77ef420232451279ff43ff30b563baf4"
server_key_version: "encryptionkey@0"
I20260925 15:07:49.818286 6583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:07:49.821439 6598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:49.822652 6583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:49.822785 6583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "e91045ac4eda4e99bd38f2aa6df3031c"
format_stamp: "Formatted at 2026-09-25 15:07:49 on dist-test-slave-nsrc"
server_key: "3a40186eb19e5be40cdf0b1ff5f710ee"
server_key_iv: "77ef420232451279ff43ff30b563baf4"
server_key_version: "encryptionkey@0"
I20260925 15:07:49.822945 6583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:49.852706 6583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:49.856455 6583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:49.856726 6583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:49.857664 6583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:49.858908 6583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:49.858992 6583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:49.859107 6583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:49.859165 6583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:49.887171 6583 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:38649
I20260925 15:07:49.887216 6710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:38649 every 8 connection(s)
I20260925 15:07:49.888443 6583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:49.889603 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 6583
I20260925 15:07:49.889802 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:07:49.890599 23472 external_mini_cluster.cc:1428] Setting key 106a32449bb471ce26f52135dfdd3ac4
Sep 25 15:07:49 dist-test-slave-nsrc krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348869, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:49.909497 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.891096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:34009 (local address 127.22.236.62:43825)
0925 15:07:49.891563 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.891569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.895125 (+ 3556us) server_negotiation.cc:407] Connection header received
0925 15:07:49.895186 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.895191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.895265 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.895364 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:49.896891 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.897943 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.899440 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.900023 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.900116 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.903933 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:49.903939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:49.903942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:49.903973 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:49.906443 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:49.907221 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:49.907226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:49.907229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:49.907395 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:49.907838 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:49.907841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:49.907843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:49.908126 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:49.908262 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.909058 (+ 796us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.909232 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":228,"threads_started":1}
I20260925 15:07:49.910809 6714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.891710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:34009)
0925 15:07:49.892498 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:49.892546 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:49.894771 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:49.895593 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:49.895601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:49.896125 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:49.896735 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.896753 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.898080 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.898084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.899314 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.899326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.900249 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.900257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.900547 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.901199 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:49.901226 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:49.903709 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:49.906579 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:49.906587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:49.906603 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:49.907119 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:49.907492 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:49.907496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:49.907500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:49.907747 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:49.908256 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:49.908264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:49.908691 (+ 427us) client_negotiation.cc:769] Sending connection context
0925 15:07:49.908985 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 15:07:49.909494 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":330,"threads_started":1}
I20260925 15:07:49.912307 6711 heartbeater.cc:347] Connected to a master server at 127.22.236.62:43825
I20260925 15:07:49.912640 6711 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:49.913506 6711 heartbeater.cc:511] Master 127.22.236.62:43825 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
I20260925 15:07:49.915593 6522 ts_manager.cc:194] Registered new tserver with Master: e91045ac4eda4e99bd38f2aa6df3031c (127.22.236.1:38649)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
I20260925 15:07:49.917070 6522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:34009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:07:49.941366 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43825
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:50.245332 6717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:50.245620 6717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:50.245713 6717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:50.254709 6717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:50.254825 6717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:50.254945 6717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:07:50.264819 6717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:50.266033 6717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:50.267331 6717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:50.279855 6723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:50.279884 6725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:50.280435 6722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:50.280829 6717 server_base.cc:1034] running on GCE node
I20260925 15:07:50.281412 6717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:50.282336 6717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:50.283545 6717 hybrid_clock.cc:648] HybridClock initialized: now 1790348870283495 us; error 36 us; skew 500 ppm
I20260925 15:07:50.283851 6717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:50 dist-test-slave-nsrc krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348870, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:50.286893 6717 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:07:50.288375 6717 webserver.cc:460] Webserver started at http://127.22.236.2:38467/ using document root <none> and password file <none>
I20260925 15:07:50.289033 6717 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:50.289149 6717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:50.289430 6717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:50.292328 6717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "7b59a00bbd044cacb81ae521578b2241"
format_stamp: "Formatted at 2026-09-25 15:07:50 on dist-test-slave-nsrc"
server_key: "847934c4b836b8ce2c84e02ccf9f04b9"
server_key_iv: "f09ce39182cb2a3288cea434c1830570"
server_key_version: "encryptionkey@0"
I20260925 15:07:50.293058 6717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "7b59a00bbd044cacb81ae521578b2241"
format_stamp: "Formatted at 2026-09-25 15:07:50 on dist-test-slave-nsrc"
server_key: "847934c4b836b8ce2c84e02ccf9f04b9"
server_key_iv: "f09ce39182cb2a3288cea434c1830570"
server_key_version: "encryptionkey@0"
I20260925 15:07:50.298280 6717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 15:07:50.301662 6732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:50.303005 6717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:07:50.303165 6717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "7b59a00bbd044cacb81ae521578b2241"
format_stamp: "Formatted at 2026-09-25 15:07:50 on dist-test-slave-nsrc"
server_key: "847934c4b836b8ce2c84e02ccf9f04b9"
server_key_iv: "f09ce39182cb2a3288cea434c1830570"
server_key_version: "encryptionkey@0"
I20260925 15:07:50.303387 6717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:50.335888 6717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:50.339632 6717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:50.339883 6717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:50.340870 6717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:50.342123 6717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:50.342197 6717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:50.342308 6717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:50.342357 6717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:50.371810 6717 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:41077
I20260925 15:07:50.371856 6844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:41077 every 8 connection(s)
I20260925 15:07:50.373088 6717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:50.381213 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 6717
I20260925 15:07:50.381392 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:07:50.381816 23472 external_mini_cluster.cc:1428] Setting key ae531eee921c92e406aeca06e5b52e93
Sep 25 15:07:50 dist-test-slave-nsrc krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348870, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:50.392844 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.375524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:50913 (local address 127.22.236.62:43825)
0925 15:07:50.375706 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:50.375711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:50.376808 (+ 1097us) server_negotiation.cc:407] Connection header received
0925 15:07:50.378590 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:50.378595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:50.378678 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:50.378763 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:50.380325 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.381216 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.382906 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.383476 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.383600 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.387300 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:50.387305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:50.387308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:50.387346 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:50.389852 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:50.390620 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:50.390624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:50.390626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:50.390753 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:50.391162 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:50.391167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:50.391169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:50.391421 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:50.391567 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:50.392264 (+ 697us) server_negotiation.cc:299] Negotiation successful
0925 15:07:50.392506 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:07:50.393939 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.375985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:50913)
0925 15:07:50.376661 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:50.376705 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:50.378334 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:50.379002 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:50.379011 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:50.379511 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:50.380154 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:50.380180 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.381366 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.381372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:50.382770 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:50.382782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.383670 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.383674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:50.383944 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.384635 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:50.384665 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:50.387098 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:50.389997 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:50.390007 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:50.390026 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:50.390501 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:50.390852 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:50.390857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:50.390861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:50.391054 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:50.391544 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:50.391552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:50.391882 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 15:07:50.392120 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 15:07:50.392668 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":281,"threads_started":1}
I20260925 15:07:50.395417 6845 heartbeater.cc:347] Connected to a master server at 127.22.236.62:43825
I20260925 15:07:50.395779 6845 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:50.396682 6845 heartbeater.cc:511] Master 127.22.236.62:43825 requested a full tablet report, sending...
I20260925 15:07:50.398370 6522 ts_manager.cc:194] Registered new tserver with Master: 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077)
I20260925 15:07:50.399466 6522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:50913
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:07:50.429198 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43825
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:50.735412 6850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:50.735672 6850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:50.735764 6850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:50.742970 6850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:50.743072 6850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:50.743189 6850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:07:50.752544 6850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:07:50.753679 6850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:50.754865 6850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:50.767429 6858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:50.767518 6855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:50.767510 6856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:50.767987 6850 server_base.cc:1034] running on GCE node
I20260925 15:07:50.768625 6850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:50.769464 6850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:50.770663 6850 hybrid_clock.cc:648] HybridClock initialized: now 1790348870770604 us; error 46 us; skew 500 ppm
I20260925 15:07:50.770954 6850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:50 dist-test-slave-nsrc krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348870, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:50.773921 6850 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:07:50.775290 6850 webserver.cc:460] Webserver started at http://127.22.236.3:40063/ using document root <none> and password file <none>
I20260925 15:07:50.775902 6850 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:50.775996 6850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:50.776278 6850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:50.778978 6850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "9f111690e5d44ac291833f292fdfffac"
format_stamp: "Formatted at 2026-09-25 15:07:50 on dist-test-slave-nsrc"
server_key: "2381c10d7e051d9e0f02df42bf4c2112"
server_key_iv: "6c8755b4d637fb1eee006073f7687e14"
server_key_version: "encryptionkey@0"
I20260925 15:07:50.779619 6850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "9f111690e5d44ac291833f292fdfffac"
format_stamp: "Formatted at 2026-09-25 15:07:50 on dist-test-slave-nsrc"
server_key: "2381c10d7e051d9e0f02df42bf4c2112"
server_key_iv: "6c8755b4d637fb1eee006073f7687e14"
server_key_version: "encryptionkey@0"
I20260925 15:07:50.784490 6850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 15:07:50.787694 6865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:50.789047 6850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:50.789192 6850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "9f111690e5d44ac291833f292fdfffac"
format_stamp: "Formatted at 2026-09-25 15:07:50 on dist-test-slave-nsrc"
server_key: "2381c10d7e051d9e0f02df42bf4c2112"
server_key_iv: "6c8755b4d637fb1eee006073f7687e14"
server_key_version: "encryptionkey@0"
I20260925 15:07:50.789376 6850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:50.815388 6850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:50.819096 6850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:50.819357 6850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:50.820303 6850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:50.821568 6850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:50.821643 6850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:50.821755 6850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:50.821807 6850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:50.850730 6850 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:41483
I20260925 15:07:50.850772 6977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:41483 every 8 connection(s)
I20260925 15:07:50.851935 6850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:50.859226 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 6850
I20260925 15:07:50.859391 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:07:50.859890 23472 external_mini_cluster.cc:1428] Setting key 09abeb27542f37b42528f56895660b38
Sep 25 15:07:50 dist-test-slave-nsrc krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348870, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:07:50.870645 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.854239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:40203 (local address 127.22.236.62:43825)
0925 15:07:50.854409 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:50.854414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:50.855520 (+ 1106us) server_negotiation.cc:407] Connection header received
0925 15:07:50.857315 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:50.857319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:50.857387 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:50.857470 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:50.858809 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.859568 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.861714 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.862135 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.862228 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.865540 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:50.865545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:50.865549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:50.865580 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:50.867959 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:50.868707 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:50.868711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:50.868713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:50.868829 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:50.869202 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:50.869205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:50.869207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:50.869472 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:50.869589 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:50.870213 (+ 624us) server_negotiation.cc:299] Negotiation successful
0925 15:07:50.870385 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:07:50.871598 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.854730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:40203)
0925 15:07:50.855368 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:50.855404 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:50.857080 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:50.857670 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:50.857678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:50.858150 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:50.858683 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:50.858700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.859706 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.859711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:50.861475 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:50.861491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.862343 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.862350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:50.862613 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.863216 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:50.863237 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:50.865345 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:50.868092 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:50.868100 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:50.868111 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:50.868594 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:50.868925 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:50.868928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:50.868931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:50.869115 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:50.869573 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:50.869578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:50.869925 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 15:07:50.870138 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 15:07:50.870519 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":241,"threads_started":1}
I20260925 15:07:50.872867 6978 heartbeater.cc:347] Connected to a master server at 127.22.236.62:43825
I20260925 15:07:50.873150 6978 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:50.873912 6978 heartbeater.cc:511] Master 127.22.236.62:43825 requested a full tablet report, sending...
I20260925 15:07:50.875324 6522 ts_manager.cc:194] Registered new tserver with Master: 9f111690e5d44ac291833f292fdfffac (127.22.236.3:41483)
I20260925 15:07:50.876086 6522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:40203
I20260925 15:07:50.887014 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:07:50.903223 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.891423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59732 (local address 127.22.236.62:43825)
0925 15:07:50.891601 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:50.891607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:50.891975 (+ 368us) server_negotiation.cc:407] Connection header received
0925 15:07:50.892165 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:50.892171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:50.892239 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:50.892320 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:50.893789 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.894510 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.895919 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.896379 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.896483 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.898562 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:50.898567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:50.898570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:50.898601 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:50.900787 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:50.901382 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:50.901387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:50.901389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:50.901524 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:50.901878 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:50.901882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:50.901884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:50.902184 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:50.902392 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:50.902718 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 15:07:50.902926 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:07:50.906960 6522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:50.913617 6522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:50.919757 6711 heartbeater.cc:503] Master 127.22.236.62:43825 was elected leader, sending a full tablet report...
I20260925 15:07:50.948568 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.925555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:41483 (local address 127.0.0.1:38350)
0925 15:07:50.939024 (+ 13469us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:50.939067 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:50.939283 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:50.941137 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:50.941145 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:50.941180 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:50.941657 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:50.941670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.943437 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.943442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:50.945828 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:50.945842 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.947669 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.947676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:50.947884 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.947942 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 15:07:50.948062 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 15:07:50.948267 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13209,"thread_start_us":14636,"threads_started":1}
I20260925 15:07:50.949397 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.925803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41077 (local address 127.0.0.1:45964)
0925 15:07:50.940468 (+ 14665us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:50.940495 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:50.940616 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:50.942086 (+ 1470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:50.942091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:50.942117 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:50.942462 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:50.942476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.944450 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.944456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:50.946394 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:50.946404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.948435 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.948442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:50.948692 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.948722 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:07:50.948884 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 15:07:50.949024 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14518,"mutex_wait_us":50,"thread_start_us":12834,"threads_started":1}
I20260925 15:07:50.949433 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.926082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45964 (local address 127.22.236.2:41077)
0925 15:07:50.938454 (+ 12372us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:50.938466 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:50.940744 (+ 2278us) server_negotiation.cc:407] Connection header received
0925 15:07:50.940843 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:50.940850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:50.941866 (+ 1016us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:50.942080 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:50.942654 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.944287 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.946533 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.947925 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.948063 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.948273 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:50.948942 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 15:07:50.949124 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12166,"thread_start_us":13372,"threads_started":1}
I20260925 15:07:50.949476 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.926204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38350 (local address 127.22.236.3:41483)
0925 15:07:50.940189 (+ 13985us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:50.940198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:50.940237 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:07:50.940346 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:50.940354 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:50.940626 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:50.940811 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:50.941858 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.943302 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.945993 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.947178 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.947293 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.947536 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:50.949027 (+ 1491us) server_negotiation.cc:299] Negotiation successful
0925 15:07:50.949170 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13811,"thread_start_us":13620,"threads_started":1}
I20260925 15:07:50.950012 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.926091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38649 (local address 127.0.0.1:60334)
0925 15:07:50.941667 (+ 15576us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:50.941684 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:50.941787 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:50.942624 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:50.942629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:50.942653 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:50.942992 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:50.943002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.944758 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.944762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:50.946489 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:50.946504 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.948448 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.948454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:50.948655 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.948884 (+ 229us) client_negotiation.cc:769] Sending connection context
0925 15:07:50.948955 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:07:50.949757 (+ 802us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15454,"mutex_wait_us":903,"thread_start_us":15301,"threads_started":1}
I20260925 15:07:50.950181 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.926324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.22.236.1:38649)
0925 15:07:50.942048 (+ 15724us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:50.942056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:50.942092 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:07:50.942182 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:50.942189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:50.942426 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:50.942626 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:50.943161 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.944628 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.946642 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.947955 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.948063 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.948321 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:50.949056 (+ 735us) server_negotiation.cc:299] Negotiation successful
0925 15:07:50.949217 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15571,"thread_start_us":15417,"threads_started":1}
I20260925 15:07:50.953374 6646 tablet_service.cc:1467] Processing CreateTablet for tablet 2402dd490fba4682ad92828e6d0dd2d0 (DEFAULT_TABLE table=test-table [id=8a43ec7622e4466e822c3bbdcf9cd88c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:50.953521 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 2402dd490fba4682ad92828e6d0dd2d0 (DEFAULT_TABLE table=test-table [id=8a43ec7622e4466e822c3bbdcf9cd88c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:50.953416 6913 tablet_service.cc:1467] Processing CreateTablet for tablet 2402dd490fba4682ad92828e6d0dd2d0 (DEFAULT_TABLE table=test-table [id=8a43ec7622e4466e822c3bbdcf9cd88c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:50.954567 6646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2402dd490fba4682ad92828e6d0dd2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:50.954737 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2402dd490fba4682ad92828e6d0dd2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:50.954736 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2402dd490fba4682ad92828e6d0dd2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:50.965186 6996 tablet_bootstrap.cc:492] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c: Bootstrap starting.
I20260925 15:07:50.968322 6996 tablet_bootstrap.cc:654] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:50.968628 6997 tablet_bootstrap.cc:492] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241: Bootstrap starting.
I20260925 15:07:50.969681 6996 log.cc:824] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:50.970077 6998 tablet_bootstrap.cc:492] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac: Bootstrap starting.
I20260925 15:07:50.972287 6997 tablet_bootstrap.cc:654] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:50.973141 6998 tablet_bootstrap.cc:654] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:50.973384 6997 log.cc:824] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:50.974103 6998 log.cc:824] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:50.974429 6996 tablet_bootstrap.cc:492] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c: No bootstrap required, opened a new log
I20260925 15:07:50.974689 6996 ts_tablet_manager.cc:1397] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260925 15:07:50.976244 6997 tablet_bootstrap.cc:492] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241: No bootstrap required, opened a new log
I20260925 15:07:50.976495 6997 ts_tablet_manager.cc:1397] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260925 15:07:50.982681 6998 tablet_bootstrap.cc:492] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac: No bootstrap required, opened a new log
I20260925 15:07:50.982933 6998 ts_tablet_manager.cc:1397] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260925 15:07:50.985019 6996 raft_consensus.cc:348] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } } peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } }
I20260925 15:07:50.985481 6996 raft_consensus.cc:374] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:50.985625 6996 raft_consensus.cc:729] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91045ac4eda4e99bd38f2aa6df3031c, State: Initialized, Role: FOLLOWER
I20260925 15:07:50.986303 6996 consensus_queue.cc:260] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } } peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } }
I20260925 15:07:50.986402 6997 raft_consensus.cc:348] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } } peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } }
I20260925 15:07:50.986829 6997 raft_consensus.cc:374] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:50.986987 6997 raft_consensus.cc:729] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b59a00bbd044cacb81ae521578b2241, State: Initialized, Role: FOLLOWER
I20260925 15:07:50.987794 6997 consensus_queue.cc:260] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } } peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } }
I20260925 15:07:50.989018 6845 heartbeater.cc:503] Master 127.22.236.62:43825 was elected leader, sending a full tablet report...
I20260925 15:07:50.989740 6997 ts_tablet_manager.cc:1428] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 15:07:50.991255 6996 ts_tablet_manager.cc:1428] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c: Time spent starting tablet: real 0.016s user 0.016s sys 0.002s
I20260925 15:07:50.992864 6998 raft_consensus.cc:348] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } } peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } }
I20260925 15:07:50.993965 6998 raft_consensus.cc:374] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:50.994074 6998 raft_consensus.cc:729] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f111690e5d44ac291833f292fdfffac, State: Initialized, Role: FOLLOWER
I20260925 15:07:50.994661 6998 consensus_queue.cc:260] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } } peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } }
I20260925 15:07:50.995685 6978 heartbeater.cc:503] Master 127.22.236.62:43825 was elected leader, sending a full tablet report...
I20260925 15:07:50.996282 6998 ts_tablet_manager.cc:1428] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 15:07:51.092988 7004 raft_consensus.cc:482] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:51.093259 7004 raft_consensus.cc:504] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } } peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } }
I20260925 15:07:51.094937 7004 leader_election.cc:288] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077), e91045ac4eda4e99bd38f2aa6df3031c (127.22.236.1:38649)
I20260925 15:07:51.101872 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:51.095756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:38525 (local address 127.22.236.1:38649)
0925 15:07:51.095960 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:51.095965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:51.095983 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:07:51.096055 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:51.096060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:51.096131 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:51.096241 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:51.096661 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.097638 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.099940 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.101118 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.101280 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:51.101353 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:51.101560 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 15:07:51.101665 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:07:51.102185 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:51.095349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:60405 (local address 127.22.236.2:41077)
0925 15:07:51.095734 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:51.095739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:51.096265 (+ 526us) server_negotiation.cc:407] Connection header received
0925 15:07:51.096425 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:51.096429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:51.096487 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:51.096554 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:51.097049 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.097950 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.100182 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.101343 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.101462 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:51.101527 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:51.101834 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 15:07:51.101957 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260925 15:07:51.102206 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:51.095268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38649 (local address 127.22.236.3:38525)
0925 15:07:51.095483 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:51.095512 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:51.095639 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:51.096272 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:51.096277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:51.096350 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:51.096520 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:51.096529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.097756 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.097759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:51.099770 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:51.099785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.101252 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.101256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:51.101393 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:51.101426 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:07:51.101532 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:07:51.101668 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":54,"spinlock_wait_cycles":25472}
I20260925 15:07:51.102227 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:51.095274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41077 (local address 127.22.236.3:60405)
0925 15:07:51.096164 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:51.096181 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:51.096305 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:51.096588 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:51.096592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:51.096616 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:51.096894 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:51.096901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.098075 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.098079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:51.100014 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:51.100028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.101526 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.101532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:51.101689 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:51.101716 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 15:07:51.101805 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:07:51.101920 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":64,"thread_start_us":501,"threads_started":1}
W20260925 15:07:51.103374 6979 tablet.cc:2369] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:07:51.103178 6666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2402dd490fba4682ad92828e6d0dd2d0" candidate_uuid: "9f111690e5d44ac291833f292fdfffac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" is_pre_election: true
I20260925 15:07:51.103327 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2402dd490fba4682ad92828e6d0dd2d0" candidate_uuid: "9f111690e5d44ac291833f292fdfffac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b59a00bbd044cacb81ae521578b2241" is_pre_election: true
I20260925 15:07:51.103762 6800 raft_consensus.cc:2454] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f111690e5d44ac291833f292fdfffac in term 0.
I20260925 15:07:51.103770 6666 raft_consensus.cc:2454] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f111690e5d44ac291833f292fdfffac in term 0.
I20260925 15:07:51.104429 6868 leader_election.cc:302] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7b59a00bbd044cacb81ae521578b2241, 9f111690e5d44ac291833f292fdfffac, e91045ac4eda4e99bd38f2aa6df3031c; no voters:
I20260925 15:07:51.104815 7004 raft_consensus.cc:2781] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:51.104936 7004 raft_consensus.cc:482] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:51.105039 7004 raft_consensus.cc:3037] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:51.107180 7004 raft_consensus.cc:504] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } } peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } }
I20260925 15:07:51.107734 7004 leader_election.cc:288] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [CANDIDATE]: Term 1 election: Requested vote from peers 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077), e91045ac4eda4e99bd38f2aa6df3031c (127.22.236.1:38649)
I20260925 15:07:51.108170 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2402dd490fba4682ad92828e6d0dd2d0" candidate_uuid: "9f111690e5d44ac291833f292fdfffac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b59a00bbd044cacb81ae521578b2241"
I20260925 15:07:51.108196 6666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2402dd490fba4682ad92828e6d0dd2d0" candidate_uuid: "9f111690e5d44ac291833f292fdfffac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91045ac4eda4e99bd38f2aa6df3031c"
I20260925 15:07:51.108352 6800 raft_consensus.cc:3037] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:51.108366 6666 raft_consensus.cc:3037] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:51.110416 6666 raft_consensus.cc:2454] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f111690e5d44ac291833f292fdfffac in term 1.
I20260925 15:07:51.110420 6800 raft_consensus.cc:2454] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f111690e5d44ac291833f292fdfffac in term 1.
I20260925 15:07:51.110772 6869 leader_election.cc:302] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f111690e5d44ac291833f292fdfffac, e91045ac4eda4e99bd38f2aa6df3031c; no voters:
I20260925 15:07:51.111025 7004 raft_consensus.cc:2781] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:51.111435 7004 raft_consensus.cc:686] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [term 1 LEADER]: Becoming Leader. State: Replica: 9f111690e5d44ac291833f292fdfffac, State: Running, Role: LEADER
I20260925 15:07:51.111994 7004 consensus_queue.cc:237] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } } peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } }
I20260925 15:07:51.116060 6521 catalog_manager.cc:5495] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f111690e5d44ac291833f292fdfffac (127.22.236.3). New cstate: current_term: 1 leader_uuid: "9f111690e5d44ac291833f292fdfffac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } health_report { overall_health: UNKNOWN } } }
W20260925 15:07:51.124939 6846 tablet.cc:2369] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:07:51.140909 6712 tablet.cc:2369] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:07:51.149178 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:51.144340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.22.236.3:41483)
0925 15:07:51.144501 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:51.144505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:51.144524 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:07:51.144608 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:51.144612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:51.144685 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:51.144817 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:51.145104 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.145997 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.147326 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.147717 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.147848 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:51.148240 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:51.148727 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:51.148873 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:51.148916 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 15:07:51.148982 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:07:51.156957 6800 raft_consensus.cc:1261] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241 [term 1 FOLLOWER]: Refusing update from remote peer 9f111690e5d44ac291833f292fdfffac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:51.156977 6666 raft_consensus.cc:1261] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Refusing update from remote peer 9f111690e5d44ac291833f292fdfffac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:51.157866 7006 consensus_queue.cc:1035] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:51.158182 7004 consensus_queue.cc:1035] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:51.170884 7011 mvcc.cc:205] Tried to move back new op lower bound from 7333268976249479168 to 7333268976078069760. Current Snapshot: MvccSnapshot[applied={T|T < 7333268976249479168}]
I20260925 15:07:51.184379 6521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:51.184648 6521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:51.187899 6521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Sending DeleteTablet for 3 replicas of tablet 2402dd490fba4682ad92828e6d0dd2d0
I20260925 15:07:51.189000 6913 tablet_service.cc:1514] Processing DeleteTablet for tablet 2402dd490fba4682ad92828e6d0dd2d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:51 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:38350
I20260925 15:07:51.189172 6646 tablet_service.cc:1514] Processing DeleteTablet for tablet 2402dd490fba4682ad92828e6d0dd2d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:51 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:60334
I20260925 15:07:51.189239 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet 2402dd490fba4682ad92828e6d0dd2d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:51 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:45964
I20260925 15:07:51.189965 7018 tablet_replica.cc:330] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac: stopping tablet replica
I20260925 15:07:51.190337 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 6583
I20260925 15:07:51.190542 7018 raft_consensus.cc:2229] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:51.197961 7018 raft_consensus.cc:2258] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:51.199250 7020 tablet_replica.cc:330] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241: stopping tablet replica
I20260925 15:07:51.199723 7020 raft_consensus.cc:2229] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:51.200299 7020 raft_consensus.cc:2258] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:51.202515 7020 ts_tablet_manager.cc:1905] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:51.202677 7018 ts_tablet_manager.cc:1905] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:51.207124 7019 tablet_replica.cc:330] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c: stopping tablet replica
I20260925 15:07:51.207563 7019 raft_consensus.cc:2229] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:51.208119 7019 raft_consensus.cc:2258] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:51.209154 7018 ts_tablet_manager.cc:1918] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:51.209307 7018 log.cc:1196] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/2402dd490fba4682ad92828e6d0dd2d0
I20260925 15:07:51.209748 7018 ts_tablet_manager.cc:1939] T 2402dd490fba4682ad92828e6d0dd2d0 P 9f111690e5d44ac291833f292fdfffac: Deleting consensus metadata
I20260925 15:07:51.211181 6507 catalog_manager.cc:4841] TS 9f111690e5d44ac291833f292fdfffac (127.22.236.3:41483): tablet 2402dd490fba4682ad92828e6d0dd2d0 (table test-table [id=8a43ec7622e4466e822c3bbdcf9cd88c]) successfully deleted
I20260925 15:07:51.211253 7020 ts_tablet_manager.cc:1918] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:51.211398 7020 log.cc:1196] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/2402dd490fba4682ad92828e6d0dd2d0
I20260925 15:07:51.211886 7020 ts_tablet_manager.cc:1939] T 2402dd490fba4682ad92828e6d0dd2d0 P 7b59a00bbd044cacb81ae521578b2241: Deleting consensus metadata
I20260925 15:07:51.213488 7019 ts_tablet_manager.cc:1905] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:51.213570 6508 catalog_manager.cc:4841] TS 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077): tablet 2402dd490fba4682ad92828e6d0dd2d0 (table test-table [id=8a43ec7622e4466e822c3bbdcf9cd88c]) successfully deleted
I20260925 15:07:51.217952 7019 ts_tablet_manager.cc:1918] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:51.218086 7019 log.cc:1196] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/2402dd490fba4682ad92828e6d0dd2d0
I20260925 15:07:51.218465 7019 ts_tablet_manager.cc:1939] T 2402dd490fba4682ad92828e6d0dd2d0 P e91045ac4eda4e99bd38f2aa6df3031c: Deleting consensus metadata
Sep 25 15:07:51 dist-test-slave-nsrc krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:07:51.220171 6509 catalog_manager.cc:4841] TS e91045ac4eda4e99bd38f2aa6df3031c (127.22.236.1:38649): tablet 2402dd490fba4682ad92828e6d0dd2d0 (table test-table [id=8a43ec7622e4466e822c3bbdcf9cd88c]) successfully deleted
I20260925 15:07:51.225996 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:51.191567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.22.236.1:38649)
0925 15:07:51.191748 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:51.191755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:51.200896 (+ 9141us) server_negotiation.cc:407] Connection header received
0925 15:07:51.201140 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:51.201146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:51.201232 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:51.201310 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:51.203265 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.204419 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.206400 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.207089 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.207189 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:51.220571 (+ 13382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:51.220582 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:51.220596 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:51.220636 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:51.223203 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:51.223913 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:51.223920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:51.223922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:51.224048 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:51.224422 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:51.224426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:51.224428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:51.224770 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:51.225062 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:51.225384 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 15:07:51.225603 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:07:51.226584 6706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:51.375528 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 6583
I20260925 15:07:51.397300 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 6717
Sep 25 15:07:51 dist-test-slave-nsrc krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:07:51.412132 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:51.398072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.22.236.2:41077)
0925 15:07:51.398238 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:51.398245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:51.398328 (+ 83us) server_negotiation.cc:407] Connection header received
0925 15:07:51.398496 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:51.398516 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:51.398581 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:51.398662 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:51.399965 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.400747 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.402101 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.402478 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.402587 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:51.406753 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:51.406767 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:51.406783 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:51.406822 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:51.409582 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:51.410203 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:51.410210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:51.410212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:51.410330 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:51.410654 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:51.410658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:51.410660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:51.411039 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:51.411311 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:51.411600 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 15:07:51.411811 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:07:51.412721 6840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:51.500102 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 6717
I20260925 15:07:51.514159 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 6850
Sep 25 15:07:51 dist-test-slave-nsrc krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:07:51.531842 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:51.515074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.22.236.3:41483)
0925 15:07:51.515322 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:51.515329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:51.515352 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:07:51.515481 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:51.515484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:51.515558 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:51.515637 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:51.517337 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.518126 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.520039 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.520542 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.520636 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:51.526052 (+ 5416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:51.526066 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:51.526081 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:51.526119 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:51.529104 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:51.529880 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:51.529886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:51.529888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:51.530050 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:51.530433 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:51.530436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:51.530438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:51.530791 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:51.531026 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:51.531337 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 15:07:51.531494 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:07:51.532478 6973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:51.652716 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 6850
I20260925 15:07:51.668111 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 6490
I20260925 15:07:51.669183 6552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:51.775731 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 6490
I20260925 15:08:11.793113 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:43825
--webserver_interface=127.22.236.62
--webserver_port=43865
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:43825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:08:12.135126 7026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:12.135390 7026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:12.135517 7026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:12.142581 7026 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:08:12.142669 7026 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:08:12.142737 7026 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:08:12.142817 7026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:08:12.142880 7026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:12.142968 7026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:12.143020 7026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:08:12.143093 7026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:08:12.152705 7026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:43825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:43825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=43865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:12.153959 7026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:12.155254 7026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:12.175963 7032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:12.175930 7033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:12.175930 7035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:12.177116 7026 server_base.cc:1034] running on GCE node
I20260925 15:08:12.177948 7026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:12.179477 7026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:12.180691 7026 hybrid_clock.cc:648] HybridClock initialized: now 1790348892180609 us; error 67 us; skew 500 ppm
I20260925 15:08:12.181083 7026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:12 dist-test-slave-nsrc krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348892, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:12.188409 7026 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:08:12.190099 7026 webserver.cc:460] Webserver started at http://127.22.236.62:43865/ using document root <none> and password file <none>
I20260925 15:08:12.190855 7026 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:12.190972 7026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:12.197368 7026 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:08:12.200397 7042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:12.201812 7026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:08:12.201992 7026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "90ec8d211ecc4d89852cc61549df1b82"
format_stamp: "Formatted at 2026-09-25 15:07:49 on dist-test-slave-nsrc"
server_key: "615a62a6a6239d9b24603a458b33542c"
server_key_iv: "e78f100c9dbd5245f2098c59f1a6d95a"
server_key_version: "encryptionkey@0"
I20260925 15:08:12.203045 7026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:12.215628 7026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:12.219904 7026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:12.220206 7026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:12.238076 7026 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:43825
I20260925 15:08:12.238095 7093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:43825 every 8 connection(s)
I20260925 15:08:12.239602 7026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:08:12.246003 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 7026
I20260925 15:08:12.246996 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:38649
--local_ip_for_outbound_sockets=127.22.236.1
--tserver_master_addrs=127.22.236.62:43825
--webserver_port=44327
--webserver_interface=127.22.236.1
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:08:12.247040 7094 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:08:12.251510 7094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Bootstrap starting.
I20260925 15:08:12.271363 7094 log.cc:824] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:12.283209 7094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:08:12.283735 7094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Bootstrap complete.
I20260925 15:08:12.292713 7094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } }
I20260925 15:08:12.293519 7094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90ec8d211ecc4d89852cc61549df1b82, State: Initialized, Role: FOLLOWER
I20260925 15:08:12.294116 7094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } }
I20260925 15:08:12.294322 7094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:08:12.294430 7094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:08:12.294592 7094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:08:12.298288 7094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } }
I20260925 15:08:12.298777 7094 leader_election.cc:302] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90ec8d211ecc4d89852cc61549df1b82; no voters:
I20260925 15:08:12.299350 7094 leader_election.cc:288] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:08:12.299487 7098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:08:12.300556 7098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 2 LEADER]: Becoming Leader. State: Replica: 90ec8d211ecc4d89852cc61549df1b82, State: Running, Role: LEADER
I20260925 15:08:12.301057 7098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } }
I20260925 15:08:12.301560 7094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:08:12.303400 7099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "90ec8d211ecc4d89852cc61549df1b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } } }
I20260925 15:08:12.303771 7099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: This master's current role is: LEADER
I20260925 15:08:12.308102 7100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90ec8d211ecc4d89852cc61549df1b82. Latest consensus state: current_term: 2 leader_uuid: "90ec8d211ecc4d89852cc61549df1b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } } }
I20260925 15:08:12.308512 7100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: This master's current role is: LEADER
I20260925 15:08:12.309903 7103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:08:12.318055 7103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:08:12.319481 7103 catalog_manager.cc:1259] Loaded cluster ID: bccb7940e50d4243b5fdc71dabd0aebc
I20260925 15:08:12.319581 7103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:08:12.323709 7103 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:08:12.325333 7103 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Loaded TSK: 0
I20260925 15:08:12.337771 7115 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:08:12.351217 7103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Generated new TSK 1
I20260925 15:08:12.351549 7103 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:08:12.600118 7096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:12.600450 7096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:12.600590 7096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:12.608112 7096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:12.608263 7096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:12.608430 7096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:08:12.618291 7096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:38649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=44327
--tserver_master_addrs=127.22.236.62:43825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:12.619547 7096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:12.620879 7096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:12.633445 7122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:12.633437 7121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:12.634229 7096 server_base.cc:1034] running on GCE node
W20260925 15:08:12.634564 7124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:12.635243 7096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:12.636232 7096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:12.637478 7096 hybrid_clock.cc:648] HybridClock initialized: now 1790348892637408 us; error 59 us; skew 500 ppm
I20260925 15:08:12.637795 7096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:12 dist-test-slave-nsrc krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348892, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:12.640913 7096 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:08:12.642431 7096 webserver.cc:460] Webserver started at http://127.22.236.1:44327/ using document root <none> and password file <none>
I20260925 15:08:12.643102 7096 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:12.643234 7096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:12.648602 7096 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:08:12.651264 7131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:12.652727 7096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:08:12.652876 7096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "e91045ac4eda4e99bd38f2aa6df3031c"
format_stamp: "Formatted at 2026-09-25 15:07:49 on dist-test-slave-nsrc"
server_key: "3a40186eb19e5be40cdf0b1ff5f710ee"
server_key_iv: "77ef420232451279ff43ff30b563baf4"
server_key_version: "encryptionkey@0"
I20260925 15:08:12.653833 7096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:12.677263 7096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:12.681363 7096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:12.681622 7096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:12.682638 7096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:12.683974 7096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:12.684069 7096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:12.684194 7096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:12.684266 7096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:12.714043 7096 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:38649
I20260925 15:08:12.714110 7243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:38649 every 8 connection(s)
I20260925 15:08:12.715437 7096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:08:12.721400 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 7096
I20260925 15:08:12.722005 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:41077
--local_ip_for_outbound_sockets=127.22.236.2
--tserver_master_addrs=127.22.236.62:43825
--webserver_port=38467
--webserver_interface=127.22.236.2
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:08:12 dist-test-slave-nsrc krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348892, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:08:12.739391 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:12.718176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:39215 (local address 127.22.236.62:43825)
0925 15:08:12.718794 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:12.718812 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:12.719304 (+ 492us) server_negotiation.cc:407] Connection header received
0925 15:08:12.721645 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:12.721659 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:12.722030 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:12.722389 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:12.724286 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.725548 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.727756 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.728637 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.728796 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:12.732340 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:12.732348 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:12.732362 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:12.732401 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:12.735092 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:12.735796 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:12.735803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:12.735808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:12.735964 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:12.736416 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:12.736420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:12.736422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:12.736884 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:12.737059 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:12.737820 (+ 761us) server_negotiation.cc:299] Negotiation successful
0925 15:08:12.738210 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":251,"threads_started":1}
I20260925 15:08:12.739384 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:12.718394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:39215)
0925 15:08:12.719131 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:12.719175 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:12.721054 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:12.722570 (+ 1516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:12.722580 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:12.723329 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:12.724054 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:12.724077 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.725713 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.725719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:12.727608 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:12.727623 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.728862 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.728869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:12.729117 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:12.729753 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:12.729776 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:12.732035 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:12.735229 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:12.735237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:12.735252 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:12.735658 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:12.736076 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:12.736080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:12.736084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:12.736305 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:12.737013 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:12.737021 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:12.737351 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 15:08:12.737582 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 15:08:12.738124 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":295,"threads_started":1}
I20260925 15:08:12.742238 7244 heartbeater.cc:347] Connected to a master server at 127.22.236.62:43825
I20260925 15:08:12.742517 7244 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:12.743191 7244 heartbeater.cc:511] Master 127.22.236.62:43825 requested a full tablet report, sending...
I20260925 15:08:12.745268 7059 ts_manager.cc:194] Registered new tserver with Master: e91045ac4eda4e99bd38f2aa6df3031c (127.22.236.1:38649)
I20260925 15:08:12.746721 7059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:39215
W20260925 15:08:13.037282 7248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:13.037647 7248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:13.037792 7248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:13.045055 7248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:13.045177 7248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:13.045338 7248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:08:13.054889 7248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:41077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=38467
--tserver_master_addrs=127.22.236.62:43825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:13.056120 7248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:13.057508 7248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:13.069792 7256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:13.069772 7254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:13.069798 7253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:13.070766 7248 server_base.cc:1034] running on GCE node
I20260925 15:08:13.071312 7248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:13.072212 7248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:13.073395 7248 hybrid_clock.cc:648] HybridClock initialized: now 1790348893073345 us; error 54 us; skew 500 ppm
I20260925 15:08:13.073675 7248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:13 dist-test-slave-nsrc krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348893, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:13.076587 7248 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:08:13.077891 7248 webserver.cc:460] Webserver started at http://127.22.236.2:38467/ using document root <none> and password file <none>
I20260925 15:08:13.078516 7248 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:13.078634 7248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:13.083608 7248 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:08:13.086722 7263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:13.088569 7248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 15:08:13.088733 7248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "7b59a00bbd044cacb81ae521578b2241"
format_stamp: "Formatted at 2026-09-25 15:07:50 on dist-test-slave-nsrc"
server_key: "847934c4b836b8ce2c84e02ccf9f04b9"
server_key_iv: "f09ce39182cb2a3288cea434c1830570"
server_key_version: "encryptionkey@0"
I20260925 15:08:13.090099 7248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:13.104714 7248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:13.108669 7248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:13.108897 7248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:13.109783 7248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:13.111064 7248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:13.111145 7248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:13.111244 7248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:13.111311 7248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:13.146997 7248 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:41077
I20260925 15:08:13.147099 7375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:41077 every 8 connection(s)
I20260925 15:08:13.148620 7248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:08:13.153692 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 7248
I20260925 15:08:13.154354 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:41483
--local_ip_for_outbound_sockets=127.22.236.3
--tserver_master_addrs=127.22.236.62:43825
--webserver_port=40063
--webserver_interface=127.22.236.3
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:08:13 dist-test-slave-nsrc krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348893, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:08:13.171558 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.151397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:36387 (local address 127.22.236.62:43825)
0925 15:08:13.151626 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:13.151633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:13.152923 (+ 1290us) server_negotiation.cc:407] Connection header received
0925 15:08:13.155613 (+ 2690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:13.155620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:13.155731 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:13.155811 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:13.158566 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.159656 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.161861 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.162412 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.162542 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.166051 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:13.166056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:13.166060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:13.166094 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:13.168607 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:13.169323 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:13.169329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:13.169333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:13.169464 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:13.169866 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:13.169871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:13.169875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:13.170150 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:13.170283 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:13.170954 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 15:08:13.171186 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:08:13.172614 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.151897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:36387)
0925 15:08:13.152744 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:13.152803 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:13.155273 (+ 2470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:13.156387 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:13.156399 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:13.157289 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:13.158242 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.158326 (+ 84us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.159873 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.159878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.161716 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.161728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.162648 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.162653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.162860 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.163509 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:13.163532 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:13.165810 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:13.168758 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:13.168768 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:13.168786 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:13.169202 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:13.169562 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:13.169567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:13.169571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:13.169753 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:13.170271 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:13.170279 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:13.170609 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 15:08:13.170839 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 15:08:13.171379 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":340,"threads_started":1}
I20260925 15:08:13.173947 7376 heartbeater.cc:347] Connected to a master server at 127.22.236.62:43825
I20260925 15:08:13.174276 7376 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:13.175151 7376 heartbeater.cc:511] Master 127.22.236.62:43825 requested a full tablet report, sending...
I20260925 15:08:13.176682 7059 ts_manager.cc:194] Registered new tserver with Master: 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077)
I20260925 15:08:13.177454 7059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:36387
W20260925 15:08:13.464849 7379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:13.465108 7379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:13.465237 7379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:13.472265 7379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:13.472363 7379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:13.472518 7379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:08:13.481851 7379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:41483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=40063
--tserver_master_addrs=127.22.236.62:43825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:13.483011 7379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:13.484248 7379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:13.496237 7385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:13.496393 7384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:13.496349 7387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:13.497219 7379 server_base.cc:1034] running on GCE node
I20260925 15:08:13.497741 7379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:13.498548 7379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:13.499734 7379 hybrid_clock.cc:648] HybridClock initialized: now 1790348893499683 us; error 44 us; skew 500 ppm
I20260925 15:08:13.500005 7379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:13 dist-test-slave-nsrc krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348893, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:13.502825 7379 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:08:13.504175 7379 webserver.cc:460] Webserver started at http://127.22.236.3:40063/ using document root <none> and password file <none>
I20260925 15:08:13.504810 7379 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:13.504930 7379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:13.510207 7379 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260925 15:08:13.512682 7394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:13.513882 7379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260925 15:08:13.514019 7379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "9f111690e5d44ac291833f292fdfffac"
format_stamp: "Formatted at 2026-09-25 15:07:50 on dist-test-slave-nsrc"
server_key: "2381c10d7e051d9e0f02df42bf4c2112"
server_key_iv: "6c8755b4d637fb1eee006073f7687e14"
server_key_version: "encryptionkey@0"
I20260925 15:08:13.514917 7379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:13.542405 7379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:13.546502 7379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:13.546746 7379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:13.547648 7379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:13.548899 7379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:13.548997 7379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:13.549099 7379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:13.549165 7379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:13.577956 7379 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:41483
I20260925 15:08:13.578028 7506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:41483 every 8 connection(s)
I20260925 15:08:13.579291 7379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:08:13.587941 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 7379
Sep 25 15:08:13 dist-test-slave-nsrc krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348893, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:08:13.600109 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.581723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:35775 (local address 127.22.236.62:43825)
0925 15:08:13.581922 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:13.581928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:13.583021 (+ 1093us) server_negotiation.cc:407] Connection header received
0925 15:08:13.585143 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:13.585147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:13.585212 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:13.585285 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:13.586730 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.587428 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.589635 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.590275 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.590413 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.594360 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:13.594365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:13.594370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:13.594412 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:13.597063 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:13.597897 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:13.597903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:13.597906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:13.598043 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:13.598427 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:13.598431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:13.598433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:13.598702 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:13.598842 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:13.599498 (+ 656us) server_negotiation.cc:299] Negotiation successful
0925 15:08:13.599719 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:08:13.601284 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.582212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:35775)
0925 15:08:13.582866 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:13.582903 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:13.584889 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:13.585507 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:13.585515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:13.586036 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:13.586576 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.586593 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.587574 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.587580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.589502 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.589513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.590707 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.590714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.591007 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.591736 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:13.591765 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:13.594149 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:13.597195 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:13.597206 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:13.597223 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:13.597781 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:13.598141 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:13.598144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:13.598146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:13.598311 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:13.598830 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:13.598838 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:13.599143 (+ 305us) client_negotiation.cc:769] Sending connection context
0925 15:08:13.599370 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 15:08:13.599879 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":250,"threads_started":1}
I20260925 15:08:13.602775 7507 heartbeater.cc:347] Connected to a master server at 127.22.236.62:43825
I20260925 15:08:13.603081 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.588967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.22.236.62:43825)
0925 15:08:13.589641 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:13.589646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:13.589663 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:08:13.589705 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:13.589708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:13.589772 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:13.589860 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:13.591941 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.592690 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.594326 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.595036 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.595129 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.597785 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:13.597789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:13.597792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:13.597820 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:13.600410 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:13.601139 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:13.601142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:13.601144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:13.601243 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:13.601645 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:13.601650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:13.601653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:13.601978 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:13.602221 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:13.602543 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 15:08:13.602771 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":385,"threads_started":1}
I20260925 15:08:13.603111 7507 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:13.603919 7507 heartbeater.cc:511] Master 127.22.236.62:43825 requested a full tablet report, sending...
I20260925 15:08:13.605302 7059 ts_manager.cc:194] Registered new tserver with Master: 9f111690e5d44ac291833f292fdfffac (127.22.236.3:41483)
I20260925 15:08:13.606083 7059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:35775
I20260925 15:08:13.617245 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:08:13.623265 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.618106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.22.236.62:43825)
0925 15:08:13.618295 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:13.618303 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:13.618642 (+ 339us) server_negotiation.cc:407] Connection header received
0925 15:08:13.618830 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:13.618834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:13.618963 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:13.619093 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:08:13.619508 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.620218 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.621608 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.622008 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.622093 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.622517 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:08:13.622717 (+ 200us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 15:08:13.622982 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47994: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 15:08:13.623394 7511 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47994: token expired
W20260925 15:08:13.623868 6982 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.22.236.62:43825)
I20260925 15:08:13.634951 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.624300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47996 (local address 127.22.236.62:43825)
0925 15:08:13.624452 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:13.624462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:13.624479 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:08:13.624617 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:13.624620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:13.624671 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:13.624738 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:13.626023 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.626733 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.627998 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.628414 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.628507 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.630696 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:13.630700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:13.630704 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:13.630733 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:13.632948 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:13.633477 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:13.633480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:13.633483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:13.633595 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:13.633846 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:13.633849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:13.633850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:13.634123 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:13.634297 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:13.634510 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 15:08:13.634660 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:08:13.638113 7059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:08:13.645169 7059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:08:13.680135 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.656205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.22.236.2:41077)
0925 15:08:13.672185 (+ 15980us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:13.672194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:13.672231 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:08:13.672334 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:13.672342 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:13.672614 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:13.672806 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:13.673745 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.675180 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.677361 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.678750 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.679409 (+ 659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.679608 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:13.679673 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 15:08:13.679833 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15803,"thread_start_us":16849,"threads_started":1}
I20260925 15:08:13.680188 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.655804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41077 (local address 127.0.0.1:55390)
0925 15:08:13.670059 (+ 14255us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:13.670106 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:13.670266 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:13.673045 (+ 2779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:13.673053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:13.673088 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:13.673465 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.673490 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.675351 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.675356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.677211 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.677223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.679213 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.679220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.679428 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.679498 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 15:08:13.679613 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 15:08:13.679792 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14014,"mutex_wait_us":19,"thread_start_us":16515,"threads_started":1}
I20260925 15:08:13.681754 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.656386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:41483 (local address 127.0.0.1:56148)
0925 15:08:13.674091 (+ 17705us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:13.674107 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:13.674204 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:13.675620 (+ 1416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:13.675625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:13.675653 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:13.676082 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.676094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.677821 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.677826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.679564 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.679577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.680993 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.680999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.681160 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.681193 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:08:13.681278 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 15:08:13.681395 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17599,"mutex_wait_us":50,"thread_start_us":13284,"threads_started":1}
I20260925 15:08:13.681802 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.656946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.22.236.3:41483)
0925 15:08:13.675079 (+ 18133us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:13.675085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:13.675111 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:08:13.675199 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:13.675205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:13.675415 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:13.675607 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:13.676278 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.677701 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.679711 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.680778 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.680896 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.681093 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:13.681327 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 15:08:13.681489 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17991,"thread_start_us":17815,"threads_started":1}
I20260925 15:08:13.683314 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.656007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38649 (local address 127.0.0.1:57830)
0925 15:08:13.673513 (+ 17506us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:13.673539 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:13.673638 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:13.675184 (+ 1546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:13.675189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:13.675216 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:13.675702 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.675724 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.677541 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.677546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.680830 (+ 3284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.680843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.682586 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.682592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.682743 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.682771 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:08:13.682861 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:08:13.682987 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17381,"mutex_wait_us":32,"thread_start_us":17846,"threads_started":1}
I20260925 15:08:13.683378 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.656552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57830 (local address 127.22.236.1:38649)
0925 15:08:13.674675 (+ 18123us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:13.674684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:13.674709 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:08:13.674793 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:13.674800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:13.675008 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:13.675182 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:13.675920 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.677411 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.681009 (+ 3598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.682382 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.682514 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.682688 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:13.682912 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 15:08:13.683075 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17967,"thread_start_us":17829,"threads_started":1}
I20260925 15:08:13.684166 7311 tablet_service.cc:1467] Processing CreateTablet for tablet 9dad5509d20d48dcaa959f21703eabb9 (DEFAULT_TABLE table=test-table [id=eceaf518f39d4a4c95b97c46d5e5923a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:08:13.685621 7311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dad5509d20d48dcaa959f21703eabb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:13.685303 7442 tablet_service.cc:1467] Processing CreateTablet for tablet 9dad5509d20d48dcaa959f21703eabb9 (DEFAULT_TABLE table=test-table [id=eceaf518f39d4a4c95b97c46d5e5923a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:08:13.686445 7442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dad5509d20d48dcaa959f21703eabb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:13.686580 7179 tablet_service.cc:1467] Processing CreateTablet for tablet 9dad5509d20d48dcaa959f21703eabb9 (DEFAULT_TABLE table=test-table [id=eceaf518f39d4a4c95b97c46d5e5923a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:08:13.687773 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dad5509d20d48dcaa959f21703eabb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:13.698112 7523 tablet_bootstrap.cc:492] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac: Bootstrap starting.
I20260925 15:08:13.700358 7524 tablet_bootstrap.cc:492] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241: Bootstrap starting.
I20260925 15:08:13.701618 7523 tablet_bootstrap.cc:654] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:13.701978 7525 tablet_bootstrap.cc:492] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c: Bootstrap starting.
I20260925 15:08:13.703354 7523 log.cc:824] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:13.703668 7524 tablet_bootstrap.cc:654] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:13.704733 7524 log.cc:824] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:13.704979 7525 tablet_bootstrap.cc:654] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:13.706079 7525 log.cc:824] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:13.707159 7523 tablet_bootstrap.cc:492] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac: No bootstrap required, opened a new log
I20260925 15:08:13.707453 7523 ts_tablet_manager.cc:1397] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260925 15:08:13.708832 7525 tablet_bootstrap.cc:492] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c: No bootstrap required, opened a new log
I20260925 15:08:13.709102 7525 ts_tablet_manager.cc:1397] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260925 15:08:13.710677 7524 tablet_bootstrap.cc:492] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241: No bootstrap required, opened a new log
I20260925 15:08:13.710923 7524 ts_tablet_manager.cc:1397] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260925 15:08:13.716995 7525 raft_consensus.cc:348] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:13.717500 7525 raft_consensus.cc:374] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:13.717592 7525 raft_consensus.cc:729] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91045ac4eda4e99bd38f2aa6df3031c, State: Initialized, Role: FOLLOWER
I20260925 15:08:13.717471 7523 raft_consensus.cc:348] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:13.717854 7523 raft_consensus.cc:374] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:13.717938 7523 raft_consensus.cc:729] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f111690e5d44ac291833f292fdfffac, State: Initialized, Role: FOLLOWER
I20260925 15:08:13.718254 7525 consensus_queue.cc:260] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:13.719050 7523 consensus_queue.cc:260] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:13.721251 7524 raft_consensus.cc:348] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:13.721676 7524 raft_consensus.cc:374] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:13.721776 7524 raft_consensus.cc:729] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b59a00bbd044cacb81ae521578b2241, State: Initialized, Role: FOLLOWER
I20260925 15:08:13.722448 7524 consensus_queue.cc:260] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:13.726390 7244 heartbeater.cc:503] Master 127.22.236.62:43825 was elected leader, sending a full tablet report...
I20260925 15:08:13.727321 7525 ts_tablet_manager.cc:1428] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c: Time spent starting tablet: real 0.018s user 0.010s sys 0.007s
I20260925 15:08:13.728770 7523 ts_tablet_manager.cc:1428] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac: Time spent starting tablet: real 0.021s user 0.013s sys 0.006s
I20260925 15:08:13.729099 7507 heartbeater.cc:503] Master 127.22.236.62:43825 was elected leader, sending a full tablet report...
I20260925 15:08:13.730216 7524 ts_tablet_manager.cc:1428] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241: Time spent starting tablet: real 0.019s user 0.008s sys 0.012s
I20260925 15:08:13.730522 7376 heartbeater.cc:503] Master 127.22.236.62:43825 was elected leader, sending a full tablet report...
W20260925 15:08:13.830853 7508 tablet.cc:2369] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:08:13.900548 7377 tablet.cc:2369] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:08:13.967633 7245 tablet.cc:2369] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:08:13.979697 7530 raft_consensus.cc:482] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:08:13.980002 7530 raft_consensus.cc:504] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:13.981637 7530 leader_election.cc:288] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077), e91045ac4eda4e99bd38f2aa6df3031c (127.22.236.1:38649)
I20260925 15:08:13.987820 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.982080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:49685 (local address 127.22.236.2:41077)
0925 15:08:13.982273 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:13.982279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:13.982370 (+ 91us) server_negotiation.cc:407] Connection header received
0925 15:08:13.982555 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:13.982561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:13.982639 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:13.982725 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:13.983143 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.984042 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.985957 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.987006 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.987113 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.987162 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:13.987544 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 15:08:13.987630 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:08:13.987920 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.982076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41077 (local address 127.22.236.3:49685)
0925 15:08:13.982254 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:13.982278 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:13.982420 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:13.982783 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:13.982787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:13.982821 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:13.982990 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.982998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.984201 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.984207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.985804 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.985817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.987180 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.987187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.987391 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.987426 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:08:13.987523 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:08:13.987653 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":1}
I20260925 15:08:13.988847 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.982080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:54539 (local address 127.22.236.1:38649)
0925 15:08:13.982273 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:13.982279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:13.983065 (+ 786us) server_negotiation.cc:407] Connection header received
0925 15:08:13.983183 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:13.983187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:13.983243 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:13.983307 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:13.983892 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.984917 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.986796 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.988009 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.988176 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.988244 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:13.988533 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 15:08:13.988633 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:08:13.988925 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.982076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38649 (local address 127.22.236.3:54539)
0925 15:08:13.982966 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:13.982981 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:13.983074 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:13.983337 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:13.983342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:13.983366 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:13.983699 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.983710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.985041 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.985045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.986647 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.986661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.988222 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.988228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.988375 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.988407 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:08:13.988491 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 15:08:13.988600 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":223,"thread_start_us":339,"threads_started":1}
I20260925 15:08:13.988983 7331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad5509d20d48dcaa959f21703eabb9" candidate_uuid: "9f111690e5d44ac291833f292fdfffac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b59a00bbd044cacb81ae521578b2241" is_pre_election: true
I20260925 15:08:13.989571 7331 raft_consensus.cc:2454] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f111690e5d44ac291833f292fdfffac in term 0.
I20260925 15:08:13.989943 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad5509d20d48dcaa959f21703eabb9" candidate_uuid: "9f111690e5d44ac291833f292fdfffac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" is_pre_election: true
I20260925 15:08:13.990219 7397 leader_election.cc:302] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b59a00bbd044cacb81ae521578b2241, 9f111690e5d44ac291833f292fdfffac; no voters:
I20260925 15:08:13.990463 7199 raft_consensus.cc:2454] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f111690e5d44ac291833f292fdfffac in term 0.
I20260925 15:08:13.990590 7530 raft_consensus.cc:2781] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:08:13.990728 7530 raft_consensus.cc:482] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:08:13.990819 7530 raft_consensus.cc:3037] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:13.992951 7530 raft_consensus.cc:504] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:13.993525 7530 leader_election.cc:288] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [CANDIDATE]: Term 1 election: Requested vote from peers 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077), e91045ac4eda4e99bd38f2aa6df3031c (127.22.236.1:38649)
I20260925 15:08:13.993924 7331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad5509d20d48dcaa959f21703eabb9" candidate_uuid: "9f111690e5d44ac291833f292fdfffac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b59a00bbd044cacb81ae521578b2241"
I20260925 15:08:13.994009 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad5509d20d48dcaa959f21703eabb9" candidate_uuid: "9f111690e5d44ac291833f292fdfffac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91045ac4eda4e99bd38f2aa6df3031c"
I20260925 15:08:13.994130 7331 raft_consensus.cc:3037] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:13.994181 7199 raft_consensus.cc:3037] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:13.997159 7331 raft_consensus.cc:2454] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f111690e5d44ac291833f292fdfffac in term 1.
I20260925 15:08:13.997159 7199 raft_consensus.cc:2454] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f111690e5d44ac291833f292fdfffac in term 1.
I20260925 15:08:13.997541 7398 leader_election.cc:302] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f111690e5d44ac291833f292fdfffac, e91045ac4eda4e99bd38f2aa6df3031c; no voters:
I20260925 15:08:13.997836 7530 raft_consensus.cc:2781] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:08:13.998230 7530 raft_consensus.cc:686] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [term 1 LEADER]: Becoming Leader. State: Replica: 9f111690e5d44ac291833f292fdfffac, State: Running, Role: LEADER
I20260925 15:08:13.998672 7530 consensus_queue.cc:237] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:14.003216 7059 catalog_manager.cc:5495] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f111690e5d44ac291833f292fdfffac (127.22.236.3). New cstate: current_term: 1 leader_uuid: "9f111690e5d44ac291833f292fdfffac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } health_report { overall_health: HEALTHY } } }
I20260925 15:08:14.027920 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.023280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.22.236.3:41483)
0925 15:08:14.023420 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:14.023431 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:14.023463 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:08:14.023650 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:14.023654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:14.023725 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:14.023834 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:08:14.024102 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:14.024841 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:14.026163 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:14.026536 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:14.026618 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:14.026985 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:08:14.027485 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:08:14.027585 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:14.027624 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 15:08:14.027684 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:08:14.035049 7331 raft_consensus.cc:1261] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241 [term 1 FOLLOWER]: Refusing update from remote peer 9f111690e5d44ac291833f292fdfffac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:08:14.035050 7199 raft_consensus.cc:1261] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Refusing update from remote peer 9f111690e5d44ac291833f292fdfffac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:08:14.035892 7530 consensus_queue.cc:1035] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:08:14.036185 7533 consensus_queue.cc:1035] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:08:14.061889 7058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:08:14.062218 7058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:08:14.065812 7058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Sending DeleteTablet for 3 replicas of tablet 9dad5509d20d48dcaa959f21703eabb9
I20260925 15:08:14.066898 7311 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dad5509d20d48dcaa959f21703eabb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:08:14 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:55390
I20260925 15:08:14.066910 7179 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dad5509d20d48dcaa959f21703eabb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:08:14 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:57830
I20260925 15:08:14.067210 7442 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dad5509d20d48dcaa959f21703eabb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:08:14 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:56148
I20260925 15:08:14.068066 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 7096
I20260925 15:08:14.084718 7547 tablet_replica.cc:330] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac: stopping tablet replica
I20260925 15:08:14.085209 7547 raft_consensus.cc:2229] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:08:14.085960 7547 raft_consensus.cc:2258] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:08:14.087898 7547 ts_tablet_manager.cc:1905] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:08:14.092165 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.069120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.22.236.1:38649)
0925 15:08:14.069268 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:14.069273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:14.069300 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:08:14.069412 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:14.069417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:14.069527 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:14.069620 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:14.071677 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:14.072865 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:14.075037 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:14.075722 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:14.075835 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:14.082266 (+ 6431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:14.082286 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:14.082307 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:14.082354 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:14.086978 (+ 4624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:14.087877 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:14.087886 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:14.087889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:14.088052 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:14.089256 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:14.089261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:14.089265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:14.089815 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:14.090078 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:14.091597 (+ 1519us) server_negotiation.cc:299] Negotiation successful
0925 15:08:14.091802 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:08:14.092871 7239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:14.095836 7547 ts_tablet_manager.cc:1918] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:08:14.096011 7547 log.cc:1196] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/9dad5509d20d48dcaa959f21703eabb9
I20260925 15:08:14.096562 7547 ts_tablet_manager.cc:1939] T 9dad5509d20d48dcaa959f21703eabb9 P 9f111690e5d44ac291833f292fdfffac: Deleting consensus metadata
I20260925 15:08:14.098439 7044 catalog_manager.cc:4841] TS 9f111690e5d44ac291833f292fdfffac (127.22.236.3:41483): tablet 9dad5509d20d48dcaa959f21703eabb9 (table test-table [id=eceaf518f39d4a4c95b97c46d5e5923a]) successfully deleted
I20260925 15:08:14.102845 7545 tablet_replica.cc:330] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241: stopping tablet replica
I20260925 15:08:14.103307 7545 raft_consensus.cc:2229] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:08:14.103812 7545 raft_consensus.cc:2258] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:08:14.105914 7545 ts_tablet_manager.cc:1905] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:08:14.113541 7545 ts_tablet_manager.cc:1918] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:08:14.113687 7545 log.cc:1196] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/9dad5509d20d48dcaa959f21703eabb9
I20260925 15:08:14.114085 7545 ts_tablet_manager.cc:1939] T 9dad5509d20d48dcaa959f21703eabb9 P 7b59a00bbd044cacb81ae521578b2241: Deleting consensus metadata
I20260925 15:08:14.115530 7045 catalog_manager.cc:4841] TS 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077): tablet 9dad5509d20d48dcaa959f21703eabb9 (table test-table [id=eceaf518f39d4a4c95b97c46d5e5923a]) successfully deleted
I20260925 15:08:14.229866 7546 tablet_replica.cc:330] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c: stopping tablet replica
I20260925 15:08:14.230105 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 7096
W20260925 15:08:14.247180 7046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.236.1:38649 (error 108)
I20260925 15:08:14.247390 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 7248
I20260925 15:08:14.262585 7550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.249140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38649 (local address 127.0.0.1:57846)
0925 15:08:14.262228 (+ 13088us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:14.262372 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12943,"thread_start_us":12851,"threads_started":1}
W20260925 15:08:14.262881 7046 catalog_manager.cc:4584] TS e91045ac4eda4e99bd38f2aa6df3031c (127.22.236.1:38649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9dad5509d20d48dcaa959f21703eabb9: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38649: connect: Connection refused (error 111)
I20260925 15:08:14.264755 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.248462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.22.236.2:41077)
0925 15:08:14.248630 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:14.248635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:14.248654 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:08:14.248718 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:14.248724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:14.248812 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:14.248912 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:14.250699 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:14.251790 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:14.253814 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:14.254453 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:14.254562 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:14.257996 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:14.258014 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:14.258033 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:14.258072 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:14.261671 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:14.262557 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:14.262568 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:14.262572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:14.262738 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:14.263172 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:14.263176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:14.263179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:14.263540 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:14.263745 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:14.264251 (+ 506us) server_negotiation.cc:299] Negotiation successful
0925 15:08:14.264420 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:08:14.265484 7371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:14.316613 7550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.316039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38649 (local address 127.0.0.1:57854)
0925 15:08:14.316297 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:14.316412 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 15:08:14.317833 7550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.317383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38649 (local address 127.0.0.1:57858)
0925 15:08:14.317545 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:14.317634 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 15:08:14.353915 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 7248
I20260925 15:08:14.375880 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 7379
I20260925 15:08:14.378522 7550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.378121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38649 (local address 127.0.0.1:57860)
0925 15:08:14.378302 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:14.378383 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 15:08:14.379509 7550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.379162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38649 (local address 127.0.0.1:57872)
0925 15:08:14.379306 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:14.379389 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:08:14.392738 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.376646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.22.236.3:41483)
0925 15:08:14.377284 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:14.377291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:14.377315 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:08:14.377382 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:14.377388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:14.377486 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:14.377591 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:14.380233 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:14.381382 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:14.382676 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:14.383328 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:14.383466 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:14.385923 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:14.385939 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:14.385960 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:14.386005 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:14.389965 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:14.390604 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:14.390614 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:14.390618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:14.390833 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:14.391168 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:14.391174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:14.391177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:14.391637 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:14.391937 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:14.392095 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 15:08:14.392339 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484}
I20260925 15:08:14.393512 7502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:14.460269 7550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.459745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38649 (local address 127.0.0.1:57880)
0925 15:08:14.459982 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:14.460078 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 15:08:14.461302 7550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.460924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38649 (local address 127.0.0.1:57896)
0925 15:08:14.461111 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:14.461180 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 15:08:14.513569 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 7379
I20260925 15:08:14.530105 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 7026
I20260925 15:08:14.531281 7089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:14.645983 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 7026
I20260925 15:08:14.646039 7550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.645328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38649 (local address 127.0.0.1:57906)
0925 15:08:14.645791 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:14.645902 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313}
I20260925 15:08:34.663619 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:43825
--webserver_interface=127.22.236.62
--webserver_port=43865
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:43825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:08:34.975224 7554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:34.975489 7554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:34.975601 7554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:34.982788 7554 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:08:34.982914 7554 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:08:34.982970 7554 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:08:34.983038 7554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:08:34.983109 7554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:34.983199 7554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:34.983254 7554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:08:34.983325 7554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:08:34.993185 7554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:43825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:43825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=43865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:34.994417 7554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:34.995712 7554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:35.007051 7560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:35.007162 7559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:35.007076 7562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:35.007331 7554 server_base.cc:1034] running on GCE node
I20260925 15:08:35.008476 7554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:35.009738 7554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:35.010951 7554 hybrid_clock.cc:648] HybridClock initialized: now 1790348915010888 us; error 43 us; skew 500 ppm
I20260925 15:08:35.011252 7554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:35 dist-test-slave-nsrc krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348915, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:35.014185 7554 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:08:35.015543 7554 webserver.cc:460] Webserver started at http://127.22.236.62:43865/ using document root <none> and password file <none>
I20260925 15:08:35.016218 7554 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:35.016305 7554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:35.021351 7554 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:08:35.023917 7569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:35.025187 7554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:08:35.025319 7554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "90ec8d211ecc4d89852cc61549df1b82"
format_stamp: "Formatted at 2026-09-25 15:07:49 on dist-test-slave-nsrc"
server_key: "615a62a6a6239d9b24603a458b33542c"
server_key_iv: "e78f100c9dbd5245f2098c59f1a6d95a"
server_key_version: "encryptionkey@0"
I20260925 15:08:35.026199 7554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:35.047643 7554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:35.051358 7554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:35.051610 7554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:35.068337 7554 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:43825
I20260925 15:08:35.068367 7620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:43825 every 8 connection(s)
I20260925 15:08:35.069722 7554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:08:35.074786 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 7554
I20260925 15:08:35.075747 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:38649
--local_ip_for_outbound_sockets=127.22.236.1
--tserver_master_addrs=127.22.236.62:43825
--webserver_port=44327
--webserver_interface=127.22.236.1
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:08:35.076895 7621 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:08:35.081130 7621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Bootstrap starting.
I20260925 15:08:35.104023 7621 log.cc:824] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:35.119593 7621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:08:35.120103 7621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Bootstrap complete.
I20260925 15:08:35.129031 7621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } }
I20260925 15:08:35.129966 7621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90ec8d211ecc4d89852cc61549df1b82, State: Initialized, Role: FOLLOWER
I20260925 15:08:35.130599 7621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } }
I20260925 15:08:35.130822 7621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:08:35.130928 7621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:08:35.131078 7621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:08:35.134799 7621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } }
I20260925 15:08:35.135249 7621 leader_election.cc:302] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90ec8d211ecc4d89852cc61549df1b82; no voters:
I20260925 15:08:35.135829 7621 leader_election.cc:288] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:08:35.136130 7625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:08:35.137024 7625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [term 3 LEADER]: Becoming Leader. State: Replica: 90ec8d211ecc4d89852cc61549df1b82, State: Running, Role: LEADER
I20260925 15:08:35.137578 7625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } }
I20260925 15:08:35.137801 7621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:08:35.140163 7627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90ec8d211ecc4d89852cc61549df1b82. Latest consensus state: current_term: 3 leader_uuid: "90ec8d211ecc4d89852cc61549df1b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } } }
I20260925 15:08:35.140686 7627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: This master's current role is: LEADER
I20260925 15:08:35.140318 7626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "90ec8d211ecc4d89852cc61549df1b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ec8d211ecc4d89852cc61549df1b82" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43825 } } }
I20260925 15:08:35.141371 7626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82 [sys.catalog]: This master's current role is: LEADER
I20260925 15:08:35.149319 7633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:08:35.158422 7633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:08:35.159905 7633 catalog_manager.cc:1259] Loaded cluster ID: bccb7940e50d4243b5fdc71dabd0aebc
I20260925 15:08:35.160014 7633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:08:35.164173 7633 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:08:35.165776 7633 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Loaded TSK: 0
I20260925 15:08:35.165925 7633 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 90ec8d211ecc4d89852cc61549df1b82: Loaded TSK: 1
F20260925 15:08:35.167130 7633 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f668dbb79ad google::LogMessage::Fail() at ??:0
@ 0x7f668dbba88a google::LogMessage::SendToLog() at ??:0
@ 0x7f668dbb74de google::LogMessage::Flush() at ??:0
@ 0x7f668dbb8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f669191b7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f669190a73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f66a35e61c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f66a35e5783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f66a36712a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f66a36710c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f66a36bb5e9 std::function<>::operator()() at ??:0
@ 0x7f66a35e9cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f66a35e7c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f66a366f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f66a366f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6690803345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f669080691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6690806799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f66907e1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6693abb6db start_thread at ??:0
@ 0x7f668ca5161f clone at ??:0
W20260925 15:08:35.454418 7623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:35.454690 7623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:35.454808 7623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:35.462010 7623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:35.462123 7623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:35.462277 7623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:08:35.471964 7623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:38649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=44327
--tserver_master_addrs=127.22.236.62:43825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:35.473212 7623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:35.474421 7623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:35.486610 7668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:35.486573 7671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:35.486634 7669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:35.487792 7623 server_base.cc:1034] running on GCE node
I20260925 15:08:35.488384 7623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:35.489264 7623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:35.490463 7623 hybrid_clock.cc:648] HybridClock initialized: now 1790348915490415 us; error 32 us; skew 500 ppm
I20260925 15:08:35.490746 7623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:35 dist-test-slave-nsrc krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348915, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:35.493712 7623 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:08:35.495113 7623 webserver.cc:460] Webserver started at http://127.22.236.1:44327/ using document root <none> and password file <none>
I20260925 15:08:35.495729 7623 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:35.495824 7623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:35.500828 7623 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:08:35.503521 7678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:35.504804 7623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:08:35.504967 7623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "e91045ac4eda4e99bd38f2aa6df3031c"
format_stamp: "Formatted at 2026-09-25 15:07:49 on dist-test-slave-nsrc"
server_key: "3a40186eb19e5be40cdf0b1ff5f710ee"
server_key_iv: "77ef420232451279ff43ff30b563baf4"
server_key_version: "encryptionkey@0"
I20260925 15:08:35.505892 7623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:35.523448 7623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:35.527225 7623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:35.527473 7623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:35.528429 7623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:35.530078 7685 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 15:08:35.535439 7623 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:08:35.535558 7623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260925 15:08:35.535671 7623 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:08:35.538745 7623 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:08:35.538833 7623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260925 15:08:35.539143 7685 tablet_bootstrap.cc:492] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c: Bootstrap starting.
I20260925 15:08:35.565304 7685 log.cc:824] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:35.576774 7685 tablet_bootstrap.cc:492] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:08:35.577505 7685 tablet_bootstrap.cc:492] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c: Bootstrap complete.
I20260925 15:08:35.578761 7685 ts_tablet_manager.cc:1397] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c: Time spent bootstrapping tablet: real 0.040s user 0.021s sys 0.016s
I20260925 15:08:35.579944 7623 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:38649
I20260925 15:08:35.579967 7792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:38649 every 8 connection(s)
I20260925 15:08:35.581276 7623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:08:35.583597 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 7623
I20260925 15:08:35.584478 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:41077
--local_ip_for_outbound_sockets=127.22.236.2
--tserver_master_addrs=127.22.236.62:43825
--webserver_port=38467
--webserver_interface=127.22.236.2
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:08:35.586539 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:35.584090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:51777)
0925 15:08:35.585236 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:35.585570 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":615,"threads_started":1}
W20260925 15:08:35.587147 7793 heartbeater.cc:644] Failed to heartbeat to 127.22.236.62:43825 (0 consecutive failures): Network error: Failed to ping master at 127.22.236.62:43825: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
I20260925 15:08:35.588012 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:35.587677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:38955)
0925 15:08:35.587831 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:35.587896 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:08:35.587688 7685 raft_consensus.cc:348] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:35.588891 7685 raft_consensus.cc:729] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91045ac4eda4e99bd38f2aa6df3031c, State: Initialized, Role: FOLLOWER
I20260925 15:08:35.589370 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:35.588480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:36603)
0925 15:08:35.589081 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:35.589173 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":241,"threads_started":1}
W20260925 15:08:35.589557 7793 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:08:35.589581 7685 consensus_queue.cc:260] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:35.590941 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:35.590639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:60347)
0925 15:08:35.590777 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:35.590841 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 15:08:35.591113 7685 ts_tablet_manager.cc:1428] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260925 15:08:35.591917 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:35.591637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:47467)
0925 15:08:35.591750 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:35.591809 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 15:08:35.896171 7798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:35.896456 7798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:35.896577 7798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:35.903642 7798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:35.903765 7798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:35.903930 7798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:08:35.913604 7798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:41077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=38467
--tserver_master_addrs=127.22.236.62:43825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:35.914831 7798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:35.916114 7798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:35.928501 7803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:35.928566 7804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:35.928596 7806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:35.929561 7798 server_base.cc:1034] running on GCE node
I20260925 15:08:35.930167 7798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:35.931102 7798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:35.932312 7798 hybrid_clock.cc:648] HybridClock initialized: now 1790348915932259 us; error 35 us; skew 500 ppm
I20260925 15:08:35.932636 7798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:35 dist-test-slave-nsrc krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348915, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:35.935631 7798 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:08:35.937256 7798 webserver.cc:460] Webserver started at http://127.22.236.2:38467/ using document root <none> and password file <none>
I20260925 15:08:35.937942 7798 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:35.938066 7798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:35.943495 7798 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 15:08:35.946249 7813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:35.947589 7798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:08:35.947763 7798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "7b59a00bbd044cacb81ae521578b2241"
format_stamp: "Formatted at 2026-09-25 15:07:50 on dist-test-slave-nsrc"
server_key: "847934c4b836b8ce2c84e02ccf9f04b9"
server_key_iv: "f09ce39182cb2a3288cea434c1830570"
server_key_version: "encryptionkey@0"
I20260925 15:08:35.948777 7798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:35.981513 7798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:35.985599 7798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:35.985867 7798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:35.986874 7798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:35.988271 7798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:35.988369 7798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:35.988477 7798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:35.988546 7798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:36.018105 7798 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:41077
I20260925 15:08:36.018158 7925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:41077 every 8 connection(s)
I20260925 15:08:36.019503 7798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:08:36.024027 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:36.022323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:58229)
0925 15:08:36.023005 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:36.023233 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":247,"threads_started":1}
W20260925 15:08:36.024617 7926 heartbeater.cc:644] Failed to heartbeat to 127.22.236.62:43825 (0 consecutive failures): Network error: Failed to ping master at 127.22.236.62:43825: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
I20260925 15:08:36.025336 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:36.025067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:42187)
0925 15:08:36.025179 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:36.025242 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:08:36.026024 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:36.025747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:33117)
0925 15:08:36.025844 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:36.025928 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:08:36.026207 7926 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:08:36.027994 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 7798
I20260925 15:08:36.028566 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:41483
--local_ip_for_outbound_sockets=127.22.236.3
--tserver_master_addrs=127.22.236.62:43825
--webserver_port=40063
--webserver_interface=127.22.236.3
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:08:36.338099 7929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:36.338369 7929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:36.338500 7929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:36.345662 7929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:36.345765 7929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:36.345909 7929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:08:36.355386 7929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:41483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=40063
--tserver_master_addrs=127.22.236.62:43825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:36.356621 7929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:36.357878 7929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:36.369992 7934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:36.370043 7935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:36.370164 7937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:36.370823 7929 server_base.cc:1034] running on GCE node
I20260925 15:08:36.371340 7929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:36.372211 7929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:36.373425 7929 hybrid_clock.cc:648] HybridClock initialized: now 1790348916373355 us; error 59 us; skew 500 ppm
I20260925 15:08:36.373732 7929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:36 dist-test-slave-nsrc krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348916, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:36.376770 7929 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:08:36.378161 7929 webserver.cc:460] Webserver started at http://127.22.236.3:40063/ using document root <none> and password file <none>
I20260925 15:08:36.378823 7929 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:36.378947 7929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:36.384111 7929 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:08:36.386679 7944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:36.388017 7929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:08:36.388203 7929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "9f111690e5d44ac291833f292fdfffac"
format_stamp: "Formatted at 2026-09-25 15:07:50 on dist-test-slave-nsrc"
server_key: "2381c10d7e051d9e0f02df42bf4c2112"
server_key_iv: "6c8755b4d637fb1eee006073f7687e14"
server_key_version: "encryptionkey@0"
I20260925 15:08:36.389218 7929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:36.412099 7929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:36.416103 7929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:36.416389 7929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:36.417351 7929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:36.418656 7929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:36.418757 7929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:36.418875 7929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:36.418944 7929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:36.448566 7929 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:41483
I20260925 15:08:36.448637 8056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:41483 every 8 connection(s)
I20260925 15:08:36.449919 7929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:08:36.454699 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:36.452790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:57905)
0925 15:08:36.453577 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:36.453862 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":341,"threads_started":1}
W20260925 15:08:36.455281 8057 heartbeater.cc:644] Failed to heartbeat to 127.22.236.62:43825 (0 consecutive failures): Network error: Failed to ping master at 127.22.236.62:43825: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
I20260925 15:08:36.456040 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:36.455737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:33107)
0925 15:08:36.455876 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:36.455941 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 15:08:36.456808 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:36.456491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:60359)
0925 15:08:36.456628 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:36.456703 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 15:08:36.457031 8057 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:08:36.458500 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 7929
I20260925 15:08:36.593590 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:36.592760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:42473)
0925 15:08:36.593262 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:36.593365 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":243,"threads_started":1}
I20260925 15:08:36.802440 8062 raft_consensus.cc:482] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:08:36.802714 8062 raft_consensus.cc:504] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:36.804495 8062 leader_election.cc:288] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077), 9f111690e5d44ac291833f292fdfffac (127.22.236.3:41483)
Sep 25 15:08:36 dist-test-slave-nsrc krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348915, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
Sep 25 15:08:36 dist-test-slave-nsrc krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348915, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:08:36.825910 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:36.805118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:37225 (local address 127.22.236.3:41483)
0925 15:08:36.807020 (+ 1902us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:36.807035 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:36.807078 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:08:36.808541 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:36.808550 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:36.808935 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:36.809221 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:36.811156 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:36.812371 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:36.814516 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:36.815163 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:36.815337 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:36.819183 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:36.819191 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:36.819204 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:36.819248 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:36.822184 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:36.823061 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:36.823068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:36.823074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:36.823233 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:36.823646 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:36.823651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:36.823653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:36.824208 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:36.824367 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:36.825145 (+ 778us) server_negotiation.cc:299] Negotiation successful
0925 15:08:36.825506 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1717,"thread_start_us":1552,"threads_started":1}
I20260925 15:08:36.825917 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:36.804987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:36511 (local address 127.22.236.2:41077)
0925 15:08:36.807143 (+ 2156us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:36.807154 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:36.807183 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:08:36.808585 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:36.808594 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:36.808939 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:36.809244 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:36.811157 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:36.812380 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:36.814149 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:36.814977 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:36.815142 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:36.819122 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:36.819132 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:36.819147 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:36.819202 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:36.822289 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:36.823075 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:36.823081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:36.823087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:36.823245 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:36.823654 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:36.823659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:36.823662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:36.824213 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:36.824382 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:36.825143 (+ 761us) server_negotiation.cc:299] Negotiation successful
0925 15:08:36.825507 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2015,"thread_start_us":1915,"threads_started":1}
I20260925 15:08:36.826071 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:36.804775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:41483 (local address 127.22.236.1:37225)
0925 15:08:36.805204 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:36.805249 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:36.805855 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:36.809359 (+ 3504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:36.809370 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:36.810216 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:36.810974 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:36.810993 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:36.812547 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:36.812551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:36.814018 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:36.814030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:36.815347 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:36.815354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:36.815517 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:36.816310 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:36.816327 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:36.818893 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:36.822340 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:36.822350 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:36.822367 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:36.822929 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:36.823348 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:36.823354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:36.823357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:36.823544 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:36.824342 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:36.824350 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:36.824660 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 15:08:36.824919 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 15:08:36.825663 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":498,"spinlock_wait_cycles":6144}
I20260925 15:08:36.827003 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:36.804844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41077 (local address 127.22.236.1:36511)
0925 15:08:36.806600 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:36.806617 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:36.806726 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:36.809361 (+ 2635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:36.809371 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:36.810219 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:36.810973 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:36.810993 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:36.812553 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:36.812557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:36.814024 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:36.814038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:36.815153 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:36.815160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:36.815409 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:36.816088 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:36.816120 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:36.818687 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:36.822436 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:36.822441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:36.822445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:36.822945 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:36.823366 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:36.823369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:36.823372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:36.823556 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:36.824350 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:36.824356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:36.824660 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 15:08:36.824919 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 15:08:36.825507 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1626,"mutex_wait_us":166,"spinlock_wait_cycles":29440,"thread_start_us":1446,"threads_started":1}
W20260925 15:08:36.829594 7681 leader_election.cc:341] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077): Not found: Tablet not found: 9dad5509d20d48dcaa959f21703eabb9
W20260925 15:08:36.829840 7680 leader_election.cc:341] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f111690e5d44ac291833f292fdfffac (127.22.236.3:41483): Not found: Tablet not found: 9dad5509d20d48dcaa959f21703eabb9
I20260925 15:08:36.829968 7680 leader_election.cc:302] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e91045ac4eda4e99bd38f2aa6df3031c; no voters: 7b59a00bbd044cacb81ae521578b2241, 9f111690e5d44ac291833f292fdfffac
I20260925 15:08:36.830319 8062 raft_consensus.cc:2726] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:08:37.027426 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:37.026662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:59039)
0925 15:08:37.027067 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:37.027186 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":183,"threads_started":1}
I20260925 15:08:37.458184 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:37.457485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:44813)
0925 15:08:37.457871 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:37.457992 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":182,"threads_started":1}
I20260925 15:08:37.594990 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:37.594341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:40975)
0925 15:08:37.594755 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:37.594832 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":208,"threads_started":1}
I20260925 15:08:38.028954 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:38.028283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:47713)
0925 15:08:38.028684 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:38.028799 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":208,"threads_started":1}
I20260925 15:08:38.459694 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:38.459023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:53621)
0925 15:08:38.459444 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:38.459524 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":205,"threads_started":1}
I20260925 15:08:38.596601 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:38.595796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:54471)
0925 15:08:38.596313 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:38.596395 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":223,"threads_started":1}
I20260925 15:08:39.030417 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:39.029724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:38869)
0925 15:08:39.030168 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:39.030254 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":232,"threads_started":1}
I20260925 15:08:39.033331 8073 raft_consensus.cc:482] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:08:39.033509 8073 raft_consensus.cc:504] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:39.034094 8073 leader_election.cc:288] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077), 9f111690e5d44ac291833f292fdfffac (127.22.236.3:41483)
W20260925 15:08:39.034935 7681 leader_election.cc:341] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077): Not found: Tablet not found: 9dad5509d20d48dcaa959f21703eabb9
W20260925 15:08:39.035113 7680 leader_election.cc:341] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f111690e5d44ac291833f292fdfffac (127.22.236.3:41483): Not found: Tablet not found: 9dad5509d20d48dcaa959f21703eabb9
I20260925 15:08:39.035207 7680 leader_election.cc:302] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e91045ac4eda4e99bd38f2aa6df3031c; no voters: 7b59a00bbd044cacb81ae521578b2241, 9f111690e5d44ac291833f292fdfffac
I20260925 15:08:39.035431 8073 raft_consensus.cc:2726] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:08:39.461187 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:39.460507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:41493)
0925 15:08:39.460934 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:39.461025 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":216,"threads_started":1}
I20260925 15:08:39.598030 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:39.597382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:46249)
0925 15:08:39.597793 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:39.597870 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":206,"threads_started":1}
I20260925 15:08:40.031989 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:40.031295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:46993)
0925 15:08:40.031716 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:40.031804 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":203,"threads_started":1}
I20260925 15:08:40.462844 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:40.462058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:59283)
0925 15:08:40.462550 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:40.462649 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":225,"threads_started":1}
I20260925 15:08:40.599807 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:40.599081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:37679)
0925 15:08:40.599542 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:40.599630 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":232,"threads_started":1}
I20260925 15:08:41.033620 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:41.032914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:51261)
0925 15:08:41.033384 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:41.033469 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":248,"threads_started":1}
I20260925 15:08:41.338693 8080 raft_consensus.cc:482] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:08:41.338917 8080 raft_consensus.cc:504] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:41.339718 8080 leader_election.cc:288] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077), 9f111690e5d44ac291833f292fdfffac (127.22.236.3:41483)
W20260925 15:08:41.340677 7681 leader_election.cc:341] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077): Not found: Tablet not found: 9dad5509d20d48dcaa959f21703eabb9
W20260925 15:08:41.341216 7680 leader_election.cc:341] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f111690e5d44ac291833f292fdfffac (127.22.236.3:41483): Not found: Tablet not found: 9dad5509d20d48dcaa959f21703eabb9
I20260925 15:08:41.341352 7680 leader_election.cc:302] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e91045ac4eda4e99bd38f2aa6df3031c; no voters: 7b59a00bbd044cacb81ae521578b2241, 9f111690e5d44ac291833f292fdfffac
I20260925 15:08:41.341589 8080 raft_consensus.cc:2726] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:08:41.464699 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:41.463768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:53639)
0925 15:08:41.464452 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:41.464541 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":272,"threads_started":1}
I20260925 15:08:41.601362 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:41.600626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:41975)
0925 15:08:41.601079 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:41.601185 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":206,"threads_started":1}
I20260925 15:08:42.035230 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:42.034405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:34301)
0925 15:08:42.034938 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:42.035045 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":236,"threads_started":1}
I20260925 15:08:42.466287 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:42.465551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:60237)
0925 15:08:42.466034 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:42.466115 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":246,"threads_started":1}
I20260925 15:08:42.602926 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:42.602211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:44569)
0925 15:08:42.602657 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:42.602738 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":242,"threads_started":1}
I20260925 15:08:43.036969 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:43.036091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:57373)
0925 15:08:43.036669 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:43.036779 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":215,"threads_started":1}
I20260925 15:08:43.467864 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:43.467111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:37105)
0925 15:08:43.467576 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:43.467655 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":205,"threads_started":1}
I20260925 15:08:43.604561 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:43.603845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:53489)
0925 15:08:43.604314 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:43.604408 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":238,"threads_started":1}
I20260925 15:08:44.038450 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:44.037797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:60809)
0925 15:08:44.038178 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:44.038255 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":195,"threads_started":1}
I20260925 15:08:44.469492 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:44.468814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:37261)
0925 15:08:44.469223 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:44.469309 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":209,"threads_started":1}
I20260925 15:08:44.606132 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:44.605427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:38853)
0925 15:08:44.605858 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:44.605961 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":212,"threads_started":1}
I20260925 15:08:45.039915 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:45.039234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:46753)
0925 15:08:45.039682 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:45.039763 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":211,"threads_started":1}
I20260925 15:08:45.471177 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:45.470427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:51925)
0925 15:08:45.470906 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:45.471007 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":232,"threads_started":1}
I20260925 15:08:45.561594 8094 raft_consensus.cc:482] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:08:45.561769 8094 raft_consensus.cc:504] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:45.562320 8094 leader_election.cc:288] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077), 9f111690e5d44ac291833f292fdfffac (127.22.236.3:41483)
W20260925 15:08:45.563257 7680 leader_election.cc:341] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f111690e5d44ac291833f292fdfffac (127.22.236.3:41483): Not found: Tablet not found: 9dad5509d20d48dcaa959f21703eabb9
W20260925 15:08:45.563428 7681 leader_election.cc:341] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077): Not found: Tablet not found: 9dad5509d20d48dcaa959f21703eabb9
I20260925 15:08:45.563565 7681 leader_election.cc:302] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e91045ac4eda4e99bd38f2aa6df3031c; no voters: 7b59a00bbd044cacb81ae521578b2241, 9f111690e5d44ac291833f292fdfffac
I20260925 15:08:45.563791 8094 raft_consensus.cc:2726] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:08:45.607827 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:45.607059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:45403)
0925 15:08:45.607504 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:45.607607 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":218,"threads_started":1}
I20260925 15:08:46.041495 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:46.040803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:53635)
0925 15:08:46.041224 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:46.041321 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":228,"threads_started":1}
I20260925 15:08:46.472675 8097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:46.471988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:42715)
0925 15:08:46.472424 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:46.472502 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":239,"threads_started":1}
I20260925 15:08:46.609489 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:46.608730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:50413)
0925 15:08:46.609173 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:46.609274 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":223,"threads_started":1}
I20260925 15:08:47.042980 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:47.042280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:42893)
0925 15:08:47.042710 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:47.042802 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":203,"threads_started":1}
I20260925 15:08:47.474177 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:47.473478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:58249)
0925 15:08:47.473936 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:47.474019 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":217,"threads_started":1}
I20260925 15:08:47.610924 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:47.610273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:42441)
0925 15:08:47.610686 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:47.610764 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":231,"threads_started":1}
I20260925 15:08:48.044512 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:48.043796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:59873)
0925 15:08:48.044252 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:48.044335 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":228,"threads_started":1}
I20260925 15:08:48.382611 8103 raft_consensus.cc:482] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:08:48.382802 8103 raft_consensus.cc:504] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b59a00bbd044cacb81ae521578b2241" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41077 } } peers { permanent_uuid: "e91045ac4eda4e99bd38f2aa6df3031c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38649 } } peers { permanent_uuid: "9f111690e5d44ac291833f292fdfffac" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 41483 } }
I20260925 15:08:48.383479 8103 leader_election.cc:288] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077), 9f111690e5d44ac291833f292fdfffac (127.22.236.3:41483)
W20260925 15:08:48.384439 7681 leader_election.cc:341] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b59a00bbd044cacb81ae521578b2241 (127.22.236.2:41077): Not found: Tablet not found: 9dad5509d20d48dcaa959f21703eabb9
W20260925 15:08:48.384605 7680 leader_election.cc:341] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f111690e5d44ac291833f292fdfffac (127.22.236.3:41483): Not found: Tablet not found: 9dad5509d20d48dcaa959f21703eabb9
I20260925 15:08:48.384766 7680 leader_election.cc:302] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e91045ac4eda4e99bd38f2aa6df3031c; no voters: 7b59a00bbd044cacb81ae521578b2241, 9f111690e5d44ac291833f292fdfffac
I20260925 15:08:48.384999 8103 raft_consensus.cc:2726] T 9dad5509d20d48dcaa959f21703eabb9 P e91045ac4eda4e99bd38f2aa6df3031c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:08:48.475716 8104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:48.475042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:45687)
0925 15:08:48.475472 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:48.475551 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":210,"threads_started":1}
I20260925 15:08:48.612634 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:48.611806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:45907)
0925 15:08:48.612295 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:48.612413 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":230,"threads_started":1}
I20260925 15:08:49.046128 8106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:49.045383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:54743)
0925 15:08:49.045874 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:49.045960 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":272,"threads_started":1}
I20260925 15:08:49.477276 8107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:49.476578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:35829)
0925 15:08:49.477021 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:49.477107 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":224,"threads_started":1}
I20260925 15:08:49.614192 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:49.613469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:53955)
0925 15:08:49.613932 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:49.614017 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":221,"threads_started":1}
I20260925 15:08:50.047580 8109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:50.046904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:58087)
0925 15:08:50.047314 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:50.047396 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":204,"threads_started":1}
I20260925 15:08:50.478816 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:50.478112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:57019)
0925 15:08:50.478543 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:50.478627 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":210,"threads_started":1}
I20260925 15:08:50.616039 8111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:50.615313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.1:39799)
0925 15:08:50.615757 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:50.615864 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":200,"threads_started":1}
I20260925 15:08:51.049415 8112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:51.048728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.2:39683)
0925 15:08:51.049156 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:51.049244 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":209,"threads_started":1}
I20260925 15:08:51.462815 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 7623
I20260925 15:08:51.480718 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:51.463935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.22.236.1:38649)
0925 15:08:51.464462 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:51.464469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:51.464496 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:08:51.464560 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:51.464566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:51.464793 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:51.464938 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:51.466360 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:51.467300 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:51.468585 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:51.469155 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:51.469253 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:51.475311 (+ 6058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:51.475326 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:51.475341 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:51.475384 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:51.478019 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:51.478638 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:51.478645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:51.478647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:51.478784 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:51.479091 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:51.479095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:51.479097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:51.479475 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:51.479779 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:51.480116 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 15:08:51.480429 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":245,"threads_started":1}
I20260925 15:08:51.480739 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:51.479951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43825 (local address 127.22.236.3:57845)
0925 15:08:51.480447 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:51.480565 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.62:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":254,"threads_started":1}
I20260925 15:08:51.481976 7788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:51.569660 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 7623
I20260925 15:08:51.583427 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 7798
I20260925 15:08:51.596390 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:51.584162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.22.236.2:41077)
0925 15:08:51.584684 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:51.584689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:51.584705 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:08:51.584774 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:51.584778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:51.584836 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:51.584932 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:51.586230 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:51.587004 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:51.588230 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:51.588665 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:51.588772 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:51.591214 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:51.591219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:51.591223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:51.591257 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:51.593901 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:51.594530 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:51.594535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:51.594539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:51.594666 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:51.595012 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:51.595016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:51.595018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:51.595268 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:51.595535 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:51.595800 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 15:08:51.596016 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":255,"threads_started":1}
I20260925 15:08:51.597069 7921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:51.679481 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 7798
I20260925 15:08:51.693548 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 7929
I20260925 15:08:51.706172 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:51.694452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.22.236.3:41483)
0925 15:08:51.695024 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:51.695029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:51.695046 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:08:51.695101 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:51.695104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:51.695173 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:51.695261 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:51.696639 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:51.697431 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:51.698587 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:51.698983 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:51.699070 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:51.701478 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:51.701482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:51.701486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:51.701518 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:51.703886 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:51.704537 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:51.704540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:51.704542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:51.704669 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:51.704979 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:51.704982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:51.704984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:51.705220 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:51.705421 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:51.705661 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 15:08:51.705818 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":282,"threads_started":1}
I20260925 15:08:51.706835 8052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:51.789187 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 7929
2026-09-25T15:08:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64096 ms)
[----------] 30 tests from SecurityITest (286457 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:08:51 dist-test-slave-nsrc krb5kdc[8122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:08:51 dist-test-slave-nsrc krb5kdc[8122](info): set up 2 sockets
Sep 25 15:08:51 dist-test-slave-nsrc krb5kdc[8122](info): commencing operation
krb5kdc: starting...
W20260925 15:08:52.792119 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:08:52 dist-test-slave-nsrc krb5kdc[8122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:08:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:08:53.016464 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:37685
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:37685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:08:53.346091 8135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:53.346365 8135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:53.346493 8135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:53.353485 8135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:08:53.353591 8135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:53.353663 8135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:53.353747 8135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:08:53.353802 8135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:08:53.363205 8135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:37685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:37685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:53.364461 8135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:53.365687 8135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:53.376991 8143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:53.377154 8141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:53.377377 8135 server_base.cc:1034] running on GCE node
W20260925 15:08:53.377424 8140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:53.378520 8135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:53.380220 8135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:53.381419 8135 hybrid_clock.cc:648] HybridClock initialized: now 1790348933381368 us; error 43 us; skew 500 ppm
I20260925 15:08:53.381754 8135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:53 dist-test-slave-nsrc krb5kdc[8122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348933, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:53.384629 8135 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:08:53.386055 8135 webserver.cc:460] Webserver started at http://127.22.236.62:45495/ using document root <none> and password file <none>
I20260925 15:08:53.386706 8135 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:53.386830 8135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:53.387106 8135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:53.390080 8135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "1ed5954303a24ac5a910ef926d24cf06"
format_stamp: "Formatted at 2026-09-25 15:08:53 on dist-test-slave-nsrc"
server_key: "cb6b22620bdd1898980f81d749be7648"
server_key_iv: "bba0550c7aaaca790a14b4b821cd2743"
server_key_version: "encryptionkey@0"
I20260925 15:08:53.390799 8135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "1ed5954303a24ac5a910ef926d24cf06"
format_stamp: "Formatted at 2026-09-25 15:08:53 on dist-test-slave-nsrc"
server_key: "cb6b22620bdd1898980f81d749be7648"
server_key_iv: "bba0550c7aaaca790a14b4b821cd2743"
server_key_version: "encryptionkey@0"
I20260925 15:08:53.396011 8135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 15:08:53.399509 8150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:53.400866 8135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:08:53.401019 8135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "1ed5954303a24ac5a910ef926d24cf06"
format_stamp: "Formatted at 2026-09-25 15:08:53 on dist-test-slave-nsrc"
server_key: "cb6b22620bdd1898980f81d749be7648"
server_key_iv: "bba0550c7aaaca790a14b4b821cd2743"
server_key_version: "encryptionkey@0"
I20260925 15:08:53.401204 8135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:53.417418 8135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:53.421365 8135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:53.421626 8135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:53.437856 8135 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:37685
I20260925 15:08:53.437876 8201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:37685 every 8 connection(s)
I20260925 15:08:53.439137 8135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:08:53.442682 8202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:53.446883 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 8135
I20260925 15:08:53.447072 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:08:53.447736 23472 external_mini_cluster.cc:1428] Setting key e141084821f732b2b225abfd63945c62
I20260925 15:08:53.452936 8202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06: Bootstrap starting.
I20260925 15:08:53.456339 8202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:53.457530 8202 log.cc:824] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:53.460886 8202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06: No bootstrap required, opened a new log
Sep 25 15:08:53 dist-test-slave-nsrc krb5kdc[8122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:08:53.469611 8202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed5954303a24ac5a910ef926d24cf06" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37685 } }
I20260925 15:08:53.470002 8202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:53.470124 8202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed5954303a24ac5a910ef926d24cf06, State: Initialized, Role: FOLLOWER
I20260925 15:08:53.470685 8202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed5954303a24ac5a910ef926d24cf06" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37685 } }
I20260925 15:08:53.470867 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:53.449563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42906 (local address 127.22.236.62:37685)
0925 15:08:53.450361 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:53.450375 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:53.450413 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:08:53.452056 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:53.452078 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:53.452470 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:53.452771 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:53.454569 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:53.455816 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:53.457687 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:53.458664 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:53.458846 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:53.463545 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:53.463553 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:53.463570 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:53.463611 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:53.466684 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:53.467547 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:53.467554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:53.467560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:53.467720 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:53.468100 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:53.468104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:53.468106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:53.468685 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:53.468889 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:53.469298 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 15:08:53.469613 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":574,"thread_start_us":345,"threads_started":1}
I20260925 15:08:53.470916 8202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:08:53.471036 8202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:08:53.471232 8202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:53.474151 8202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed5954303a24ac5a910ef926d24cf06" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37685 } }
I20260925 15:08:53.474588 8202 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ed5954303a24ac5a910ef926d24cf06; no voters:
I20260925 15:08:53.475118 8202 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:08:53.475206 8207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:08:53.476281 8207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [term 1 LEADER]: Becoming Leader. State: Replica: 1ed5954303a24ac5a910ef926d24cf06, State: Running, Role: LEADER
I20260925 15:08:53.476759 8207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed5954303a24ac5a910ef926d24cf06" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37685 } }
I20260925 15:08:53.477077 8202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:08:53.479435 8208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ed5954303a24ac5a910ef926d24cf06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed5954303a24ac5a910ef926d24cf06" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37685 } } }
I20260925 15:08:53.479482 8209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ed5954303a24ac5a910ef926d24cf06. Latest consensus state: current_term: 1 leader_uuid: "1ed5954303a24ac5a910ef926d24cf06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed5954303a24ac5a910ef926d24cf06" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37685 } } }
I20260925 15:08:53.479871 8208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [sys.catalog]: This master's current role is: LEADER
I20260925 15:08:53.479871 8209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06 [sys.catalog]: This master's current role is: LEADER
I20260925 15:08:53.482151 8214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:08:53.486852 8214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:08:53.495651 8214 catalog_manager.cc:1347] Generated new cluster ID: 328abd8abd354aca8c0e5940b853eb0c
I20260925 15:08:53.495756 8214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:08:53.512478 8214 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:08:53.513903 8214 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:08:53.525144 8214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06: Generated new TSK 0
I20260925 15:08:53.526153 8214 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:08:53.583830 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37685
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:08:53.899963 8228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:53.900277 8228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:53.900430 8228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:53.907763 8228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:53.907864 8228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:53.907979 8228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:08:53.917513 8228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:53.918648 8228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:53.919878 8228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:53.932178 8234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:53.932307 8233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:53.932271 8236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:53.932969 8228 server_base.cc:1034] running on GCE node
I20260925 15:08:53.933521 8228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:53.934355 8228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:53.935546 8228 hybrid_clock.cc:648] HybridClock initialized: now 1790348933935501 us; error 41 us; skew 500 ppm
I20260925 15:08:53.935823 8228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:53 dist-test-slave-nsrc krb5kdc[8122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348933, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:53.938841 8228 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:08:53.940488 8228 webserver.cc:460] Webserver started at http://127.22.236.1:40011/ using document root <none> and password file <none>
I20260925 15:08:53.941110 8228 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:53.941219 8228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:53.941502 8228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:53.944276 8228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa"
format_stamp: "Formatted at 2026-09-25 15:08:53 on dist-test-slave-nsrc"
server_key: "5e07f5004dd4eb4cb2c48dcc69fad4e1"
server_key_iv: "014403c56010f6c7b2b09c6e5bc0eff6"
server_key_version: "encryptionkey@0"
I20260925 15:08:53.945012 8228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa"
format_stamp: "Formatted at 2026-09-25 15:08:53 on dist-test-slave-nsrc"
server_key: "5e07f5004dd4eb4cb2c48dcc69fad4e1"
server_key_iv: "014403c56010f6c7b2b09c6e5bc0eff6"
server_key_version: "encryptionkey@0"
I20260925 15:08:53.949692 8228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:08:53.952800 8243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:53.954008 8228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:08:53.954141 8228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa"
format_stamp: "Formatted at 2026-09-25 15:08:53 on dist-test-slave-nsrc"
server_key: "5e07f5004dd4eb4cb2c48dcc69fad4e1"
server_key_iv: "014403c56010f6c7b2b09c6e5bc0eff6"
server_key_version: "encryptionkey@0"
I20260925 15:08:53.954309 8228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:53.980015 8228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:53.983927 8228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:53.984220 8228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:53.985138 8228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:53.986373 8228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:53.986449 8228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:53.986565 8228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:53.986615 8228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:54.015833 8228 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:35807
I20260925 15:08:54.015877 8355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:35807 every 8 connection(s)
I20260925 15:08:54.017071 8228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:08:54.018049 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 8228
I20260925 15:08:54.018255 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:08:54.018857 23472 external_mini_cluster.cc:1428] Setting key 742ddf2a67fec16698eea7e643d0fecb
Sep 25 15:08:54 dist-test-slave-nsrc krb5kdc[8122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348933, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:08:54.037714 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:54.019594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:46785 (local address 127.22.236.62:37685)
0925 15:08:54.020058 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:54.020064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:54.021041 (+ 977us) server_negotiation.cc:407] Connection header received
0925 15:08:54.022805 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:54.022811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:54.022893 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:54.022991 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:54.024579 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:54.025694 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:54.027768 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:54.028198 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:54.028272 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:54.031975 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:54.031981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:54.031984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:54.032020 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:54.034907 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:54.035522 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:54.035527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:54.035530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:54.035672 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:54.036024 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:54.036029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:54.036032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:54.036356 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:54.036508 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:54.037104 (+ 596us) server_negotiation.cc:299] Negotiation successful
0925 15:08:54.037324 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":236,"threads_started":1}
I20260925 15:08:54.038403 8359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:54.020102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37685 (local address 127.22.236.1:46785)
0925 15:08:54.020893 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:54.020947 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:54.022564 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:54.023224 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:54.023233 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:54.023722 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:54.024404 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:54.024434 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:54.025834 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:54.025839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:54.027406 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:54.027431 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:54.028531 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:54.028535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:54.028790 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:54.029409 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:54.029442 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:54.031765 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:54.035026 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:54.035032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:54.035044 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:54.035426 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:54.035758 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:54.035761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:54.035764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:54.035938 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:54.036475 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:54.036481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:54.036764 (+ 283us) client_negotiation.cc:769] Sending connection context
0925 15:08:54.036992 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 15:08:54.037349 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":280,"threads_started":1}
I20260925 15:08:54.039830 8356 heartbeater.cc:347] Connected to a master server at 127.22.236.62:37685
I20260925 15:08:54.040207 8356 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:54.040876 8356 heartbeater.cc:511] Master 127.22.236.62:37685 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
I20260925 15:08:54.042881 8167 ts_manager.cc:194] Registered new tserver with Master: 0ff7cb54d38a436e95ebbcd1a66e70aa (127.22.236.1:35807)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
I20260925 15:08:54.044373 8167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:46785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:08:54.069717 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37685
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:08:54.378491 8362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:54.378783 8362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:54.378927 8362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:54.387281 8362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:54.387411 8362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:54.387574 8362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:08:54.397902 8362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:54.399210 8362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:54.400609 8362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:54.413128 8367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:54.413085 8368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:54.413173 8370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:54.413997 8362 server_base.cc:1034] running on GCE node
I20260925 15:08:54.414572 8362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:54.415467 8362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:54.416659 8362 hybrid_clock.cc:648] HybridClock initialized: now 1790348934416613 us; error 37 us; skew 500 ppm
I20260925 15:08:54.416956 8362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:54 dist-test-slave-nsrc krb5kdc[8122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348934, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:54.419777 8362 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:08:54.421139 8362 webserver.cc:460] Webserver started at http://127.22.236.2:36605/ using document root <none> and password file <none>
I20260925 15:08:54.421758 8362 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:54.421865 8362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:54.422160 8362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:54.425011 8362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "0b74e71ee6dd48708fdd7e03da933a9b"
format_stamp: "Formatted at 2026-09-25 15:08:54 on dist-test-slave-nsrc"
server_key: "6ba426c1f4171ef7bc72a5742623a43c"
server_key_iv: "43addd8167e68019d51b13f86ce0c581"
server_key_version: "encryptionkey@0"
I20260925 15:08:54.425725 8362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "0b74e71ee6dd48708fdd7e03da933a9b"
format_stamp: "Formatted at 2026-09-25 15:08:54 on dist-test-slave-nsrc"
server_key: "6ba426c1f4171ef7bc72a5742623a43c"
server_key_iv: "43addd8167e68019d51b13f86ce0c581"
server_key_version: "encryptionkey@0"
I20260925 15:08:54.430558 8362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:08:54.434351 8377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:54.436035 8362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:08:54.436262 8362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "0b74e71ee6dd48708fdd7e03da933a9b"
format_stamp: "Formatted at 2026-09-25 15:08:54 on dist-test-slave-nsrc"
server_key: "6ba426c1f4171ef7bc72a5742623a43c"
server_key_iv: "43addd8167e68019d51b13f86ce0c581"
server_key_version: "encryptionkey@0"
I20260925 15:08:54.436481 8362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:54.455138 8362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:54.458791 8362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:54.459026 8362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:54.459852 8362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:54.461159 8362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:54.461231 8362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:54.461323 8362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:54.461364 8362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:54.490289 8362 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:41021
I20260925 15:08:54.490337 8489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:41021 every 8 connection(s)
I20260925 15:08:54.491509 8362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:08:54.498903 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 8362
I20260925 15:08:54.499070 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:08:54.499632 23472 external_mini_cluster.cc:1428] Setting key 418e0cebde3d34dd96588f5e0c098e16
Sep 25 15:08:54 dist-test-slave-nsrc krb5kdc[8122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348934, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:08:54.511144 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:54.493773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:49199 (local address 127.22.236.62:37685)
0925 15:08:54.493936 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:54.493940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:54.495043 (+ 1103us) server_negotiation.cc:407] Connection header received
0925 15:08:54.496789 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:54.496793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:54.496863 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:54.496950 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:54.498276 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:54.498996 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:54.501055 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:54.501665 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:54.501792 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:54.505672 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:54.505677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:54.505680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:54.505713 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:54.508176 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:54.508941 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:54.508946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:54.508948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:54.509071 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:54.509487 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:54.509490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:54.509492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:54.509758 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:54.509903 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:54.510579 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 15:08:54.510807 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:08:54.512102 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:54.494203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37685 (local address 127.22.236.2:49199)
0925 15:08:54.494867 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:54.494914 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:54.496538 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:54.497120 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:54.497127 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:54.497612 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:54.498143 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:54.498160 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:54.499159 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:54.499164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:54.500918 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:54.500930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:54.501919 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:54.501927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:54.502202 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:54.502874 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:54.502917 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:54.505445 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:54.508318 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:54.508328 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:54.508346 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:54.508814 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:54.509174 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:54.509179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:54.509183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:54.509373 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:54.509878 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:54.509887 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:54.510232 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 15:08:54.510462 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 15:08:54.511000 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":259,"threads_started":1}
I20260925 15:08:54.513545 8490 heartbeater.cc:347] Connected to a master server at 127.22.236.62:37685
I20260925 15:08:54.513819 8490 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:54.514498 8490 heartbeater.cc:511] Master 127.22.236.62:37685 requested a full tablet report, sending...
I20260925 15:08:54.515851 8167 ts_manager.cc:194] Registered new tserver with Master: 0b74e71ee6dd48708fdd7e03da933a9b (127.22.236.2:41021)
I20260925 15:08:54.516674 8167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:49199
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:08:54.549086 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37685
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:08:54.856470 8495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:54.856746 8495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:54.856876 8495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:54.863979 8495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:54.864107 8495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:54.864282 8495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:08:54.873889 8495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:37685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:54.875116 8495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:54.876400 8495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:54.888732 8500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:54.888865 8501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:54.888842 8503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:54.889564 8495 server_base.cc:1034] running on GCE node
I20260925 15:08:54.890134 8495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:54.891069 8495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:54.892225 8495 hybrid_clock.cc:648] HybridClock initialized: now 1790348934892180 us; error 32 us; skew 500 ppm
I20260925 15:08:54.892529 8495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:54 dist-test-slave-nsrc krb5kdc[8122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348934, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:54.895470 8495 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:08:54.896895 8495 webserver.cc:460] Webserver started at http://127.22.236.3:40997/ using document root <none> and password file <none>
I20260925 15:08:54.897601 8495 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:54.897718 8495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:54.898001 8495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:54.900861 8495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "986f58304b8b4cb19e79246002a38148"
format_stamp: "Formatted at 2026-09-25 15:08:54 on dist-test-slave-nsrc"
server_key: "ca70002d757260413fb94d46a7cfd7cb"
server_key_iv: "03abc415cb5efa2a7a3a9103c7dc9818"
server_key_version: "encryptionkey@0"
I20260925 15:08:54.901582 8495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "986f58304b8b4cb19e79246002a38148"
format_stamp: "Formatted at 2026-09-25 15:08:54 on dist-test-slave-nsrc"
server_key: "ca70002d757260413fb94d46a7cfd7cb"
server_key_iv: "03abc415cb5efa2a7a3a9103c7dc9818"
server_key_version: "encryptionkey@0"
I20260925 15:08:54.906575 8495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:08:54.909992 8510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:54.911269 8495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:08:54.911440 8495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "986f58304b8b4cb19e79246002a38148"
format_stamp: "Formatted at 2026-09-25 15:08:54 on dist-test-slave-nsrc"
server_key: "ca70002d757260413fb94d46a7cfd7cb"
server_key_iv: "03abc415cb5efa2a7a3a9103c7dc9818"
server_key_version: "encryptionkey@0"
I20260925 15:08:54.911616 8495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:54.929373 8495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:54.933316 8495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:54.933622 8495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:54.934480 8495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:54.935751 8495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:54.935822 8495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:54.935912 8495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:54.935954 8495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:54.965216 8495 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:38903
I20260925 15:08:54.965255 8622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:38903 every 8 connection(s)
I20260925 15:08:54.966414 8495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:08:54.968876 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 8495
I20260925 15:08:54.969069 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:08:54.969591 23472 external_mini_cluster.cc:1428] Setting key e05a2a075f584a6b1593676c8de5fde1
Sep 25 15:08:54 dist-test-slave-nsrc krb5kdc[8122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348934, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:08:54.985304 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:54.968780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:41885 (local address 127.22.236.62:37685)
0925 15:08:54.968945 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:54.968950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:54.970287 (+ 1337us) server_negotiation.cc:407] Connection header received
0925 15:08:54.972270 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:54.972273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:54.972341 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:54.972426 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:54.973824 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:54.974680 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:54.976134 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:54.976558 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:54.976666 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:54.979845 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:54.979850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:54.979853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:54.979886 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:54.982222 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:54.982915 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:54.982920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:54.982923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:54.983095 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:54.983556 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:54.983560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:54.983562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:54.983893 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:54.984038 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:54.984859 (+ 821us) server_negotiation.cc:299] Negotiation successful
0925 15:08:54.985057 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:08:54.986268 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:54.969376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:37685 (local address 127.22.236.3:41885)
0925 15:08:54.970102 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:54.970155 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:54.972025 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:54.972663 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:54.972671 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:54.973136 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:54.973677 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:54.973693 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:54.974817 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:54.974820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:54.976005 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:54.976016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:54.976791 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:54.976799 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:54.977011 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:54.977619 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:54.977645 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:54.979665 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:54.982348 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:54.982355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:54.982369 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:54.982803 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:54.983202 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:54.983207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:54.983211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:54.983458 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:54.984023 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:54.984031 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:54.984495 (+ 464us) client_negotiation.cc:769] Sending connection context
0925 15:08:54.984785 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 15:08:54.985236 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":291,"threads_started":1}
I20260925 15:08:54.987422 8623 heartbeater.cc:347] Connected to a master server at 127.22.236.62:37685
I20260925 15:08:54.987687 8623 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:54.988394 8623 heartbeater.cc:511] Master 127.22.236.62:37685 requested a full tablet report, sending...
I20260925 15:08:54.989701 8167 ts_manager.cc:194] Registered new tserver with Master: 986f58304b8b4cb19e79246002a38148 (127.22.236.3:38903)
I20260925 15:08:54.990465 8167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:41885
I20260925 15:08:54.996848 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:08:55.012670 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.001103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.22.236.62:37685)
0925 15:08:55.001300 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:55.001308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:55.001655 (+ 347us) server_negotiation.cc:407] Connection header received
0925 15:08:55.001802 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:55.001806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:55.001867 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:55.001939 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:55.003253 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.003971 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.005445 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.005837 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.005933 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.008174 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:55.008180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:55.008185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:55.008216 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:55.010514 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:55.011073 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:55.011076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:55.011078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:55.011204 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:55.011468 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:55.011471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:55.011473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:55.011759 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:55.011934 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:55.012216 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 15:08:55.012376 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:08:55.016781 8167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:08:55.025127 8167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:08:55.046613 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.036189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.22.236.1:35807)
0925 15:08:55.036892 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:55.036898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:55.037544 (+ 646us) server_negotiation.cc:407] Connection header received
0925 15:08:55.037801 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:55.037809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:55.038061 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:55.038238 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:55.039158 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.040848 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.043822 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.045173 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.045300 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.045532 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:55.046126 (+ 594us) server_negotiation.cc:299] Negotiation successful
0925 15:08:55.046289 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":424,"threads_started":1}
I20260925 15:08:55.046790 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.036120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:35807 (local address 127.0.0.1:53514)
0925 15:08:55.037402 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:55.037444 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:55.037605 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:55.038524 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:55.038531 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:55.038563 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:55.038971 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:55.038984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.041100 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.041106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:55.043669 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:55.043682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.045657 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.045664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:55.045878 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.045933 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:08:55.046065 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 15:08:55.046330 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"mutex_wait_us":64,"thread_start_us":1000,"threads_started":1}
I20260925 15:08:55.047582 8356 heartbeater.cc:503] Master 127.22.236.62:37685 was elected leader, sending a full tablet report...
I20260925 15:08:55.050684 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.036178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:41021 (local address 127.0.0.1:58534)
0925 15:08:55.042666 (+ 6488us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:55.042686 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:55.042806 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:55.043684 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:55.043688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:55.043713 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:55.044024 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:55.044033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.045744 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.045753 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:55.047888 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:55.047903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.049878 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.049884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:55.050046 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.050075 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:08:55.050157 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:08:55.050288 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6363,"mutex_wait_us":23,"thread_start_us":6266,"threads_started":1}
I20260925 15:08:55.050942 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.036719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58534 (local address 127.22.236.2:41021)
0925 15:08:55.037309 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:55.037318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:55.042933 (+ 5615us) server_negotiation.cc:407] Connection header received
0925 15:08:55.043041 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:55.043048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:55.043276 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:55.043482 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:55.044191 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.045591 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.048073 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.049451 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.049577 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.049752 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:55.050482 (+ 730us) server_negotiation.cc:299] Negotiation successful
0925 15:08:55.050630 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":1808,"threads_started":1}
I20260925 15:08:55.051162 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.037990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:38903 (local address 127.0.0.1:33798)
0925 15:08:55.042986 (+ 4996us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:55.043001 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:55.043089 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:55.044264 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:55.044269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:55.044294 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:55.044619 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:55.044630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.046463 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.046471 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:55.048597 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:55.048610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.050247 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.050253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:55.050424 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.050456 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:08:55.050543 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:08:55.050665 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4866,"thread_start_us":5169,"threads_started":1}
I20260925 15:08:55.050707 8291 tablet_service.cc:1467] Processing CreateTablet for tablet be78410bbd5146b49013cf2c9ba12626 (DEFAULT_TABLE table=test-table [id=4509f05e182c4c428ee66330b0676081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:08:55.051262 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.038288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.22.236.3:38903)
0925 15:08:55.043735 (+ 5447us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:55.043742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:55.043769 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:08:55.043853 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:55.043859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:55.044064 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:55.044256 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:55.044792 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.046307 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.048776 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.050062 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.050175 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.050372 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:55.050800 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 15:08:55.050955 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5294,"thread_start_us":5139,"threads_started":1}
I20260925 15:08:55.052165 8291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be78410bbd5146b49013cf2c9ba12626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:55.053879 8425 tablet_service.cc:1467] Processing CreateTablet for tablet be78410bbd5146b49013cf2c9ba12626 (DEFAULT_TABLE table=test-table [id=4509f05e182c4c428ee66330b0676081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:08:55.055107 8425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be78410bbd5146b49013cf2c9ba12626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:55.055101 8558 tablet_service.cc:1467] Processing CreateTablet for tablet be78410bbd5146b49013cf2c9ba12626 (DEFAULT_TABLE table=test-table [id=4509f05e182c4c428ee66330b0676081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:08:55.056387 8558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be78410bbd5146b49013cf2c9ba12626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:55.065447 8641 tablet_bootstrap.cc:492] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b: Bootstrap starting.
I20260925 15:08:55.066332 8642 tablet_bootstrap.cc:492] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa: Bootstrap starting.
I20260925 15:08:55.069617 8641 tablet_bootstrap.cc:654] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:55.069617 8642 tablet_bootstrap.cc:654] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:55.069954 8643 tablet_bootstrap.cc:492] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148: Bootstrap starting.
I20260925 15:08:55.070838 8642 log.cc:824] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:55.070837 8641 log.cc:824] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:55.073431 8643 tablet_bootstrap.cc:654] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:55.073467 8642 tablet_bootstrap.cc:492] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa: No bootstrap required, opened a new log
I20260925 15:08:55.073700 8642 ts_tablet_manager.cc:1397] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 15:08:55.073854 8641 tablet_bootstrap.cc:492] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b: No bootstrap required, opened a new log
I20260925 15:08:55.074119 8641 ts_tablet_manager.cc:1397] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 15:08:55.074576 8643 log.cc:824] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:55.077464 8643 tablet_bootstrap.cc:492] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148: No bootstrap required, opened a new log
I20260925 15:08:55.077767 8643 ts_tablet_manager.cc:1397] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:08:55.081559 8641 raft_consensus.cc:348] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35807 } } peers { permanent_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41021 } } peers { permanent_uuid: "986f58304b8b4cb19e79246002a38148" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38903 } }
I20260925 15:08:55.081887 8641 raft_consensus.cc:374] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:55.082016 8641 raft_consensus.cc:729] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b74e71ee6dd48708fdd7e03da933a9b, State: Initialized, Role: FOLLOWER
I20260925 15:08:55.082559 8641 consensus_queue.cc:260] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35807 } } peers { permanent_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41021 } } peers { permanent_uuid: "986f58304b8b4cb19e79246002a38148" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38903 } }
I20260925 15:08:55.083675 8642 raft_consensus.cc:348] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35807 } } peers { permanent_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41021 } } peers { permanent_uuid: "986f58304b8b4cb19e79246002a38148" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38903 } }
I20260925 15:08:55.084164 8642 raft_consensus.cc:374] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:55.084297 8642 raft_consensus.cc:729] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff7cb54d38a436e95ebbcd1a66e70aa, State: Initialized, Role: FOLLOWER
I20260925 15:08:55.085026 8642 consensus_queue.cc:260] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35807 } } peers { permanent_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41021 } } peers { permanent_uuid: "986f58304b8b4cb19e79246002a38148" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38903 } }
I20260925 15:08:55.087752 8643 raft_consensus.cc:348] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35807 } } peers { permanent_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41021 } } peers { permanent_uuid: "986f58304b8b4cb19e79246002a38148" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38903 } }
I20260925 15:08:55.088264 8643 raft_consensus.cc:374] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:55.089205 8643 raft_consensus.cc:729] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 986f58304b8b4cb19e79246002a38148, State: Initialized, Role: FOLLOWER
I20260925 15:08:55.090052 8641 ts_tablet_manager.cc:1428] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b: Time spent starting tablet: real 0.016s user 0.010s sys 0.007s
I20260925 15:08:55.089973 8643 consensus_queue.cc:260] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35807 } } peers { permanent_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41021 } } peers { permanent_uuid: "986f58304b8b4cb19e79246002a38148" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38903 } }
I20260925 15:08:55.091249 8623 heartbeater.cc:503] Master 127.22.236.62:37685 was elected leader, sending a full tablet report...
I20260925 15:08:55.092036 8490 heartbeater.cc:503] Master 127.22.236.62:37685 was elected leader, sending a full tablet report...
I20260925 15:08:55.093515 8643 ts_tablet_manager.cc:1428] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20260925 15:08:55.094946 8642 ts_tablet_manager.cc:1428] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
W20260925 15:08:55.217892 8624 tablet.cc:2369] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:08:55.243361 8491 tablet.cc:2369] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:08:55.269373 8357 tablet.cc:2369] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:08:55.425459 8647 raft_consensus.cc:482] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:08:55.425761 8647 raft_consensus.cc:504] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35807 } } peers { permanent_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41021 } } peers { permanent_uuid: "986f58304b8b4cb19e79246002a38148" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38903 } }
I20260925 15:08:55.427547 8647 leader_election.cc:288] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ff7cb54d38a436e95ebbcd1a66e70aa (127.22.236.1:35807), 986f58304b8b4cb19e79246002a38148 (127.22.236.3:38903)
I20260925 15:08:55.434566 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.428016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:41117 (local address 127.22.236.1:35807)
0925 15:08:55.428223 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:55.428227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:55.428684 (+ 457us) server_negotiation.cc:407] Connection header received
0925 15:08:55.428898 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:55.428905 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:55.428982 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:55.429089 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:55.429687 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.430585 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.432751 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.433597 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.433721 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.433772 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:55.434192 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 15:08:55.434324 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:08:55.434674 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.428827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:58619 (local address 127.22.236.3:38903)
0925 15:08:55.428994 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:55.428999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:55.429349 (+ 350us) server_negotiation.cc:407] Connection header received
0925 15:08:55.429473 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:55.429479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:55.429551 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:55.429642 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:55.430218 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.431132 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.432960 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.433785 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.433945 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.434013 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:55.434315 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 15:08:55.434447 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:08:55.434680 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.427882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:35807 (local address 127.22.236.2:41117)
0925 15:08:55.428581 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:55.428599 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:55.428745 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:55.429126 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:55.429131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:55.429158 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:55.429534 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:55.429547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.430708 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.430714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:55.432591 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:55.432604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.433812 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.433818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:55.434028 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.434061 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:08:55.434137 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:08:55.434227 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":53,"thread_start_us":359,"threads_started":1}
I20260925 15:08:55.434783 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.428704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:38903 (local address 127.22.236.2:58619)
0925 15:08:55.429253 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:55.429266 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:55.429363 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:55.429678 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:55.429682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:55.429704 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:55.430062 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:55.430073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.431269 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.431273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:55.432804 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:55.432817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.433974 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.433979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:55.434153 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.434184 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:08:55.434269 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 15:08:55.434385 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":19,"thread_start_us":284,"threads_started":1}
I20260925 15:08:55.435787 8311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be78410bbd5146b49013cf2c9ba12626" candidate_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa" is_pre_election: true
I20260925 15:08:55.435901 8578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be78410bbd5146b49013cf2c9ba12626" candidate_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "986f58304b8b4cb19e79246002a38148" is_pre_election: true
I20260925 15:08:55.436385 8578 raft_consensus.cc:2454] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b74e71ee6dd48708fdd7e03da933a9b in term 0.
I20260925 15:08:55.436385 8311 raft_consensus.cc:2454] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b74e71ee6dd48708fdd7e03da933a9b in term 0.
I20260925 15:08:55.436998 8380 leader_election.cc:302] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b74e71ee6dd48708fdd7e03da933a9b, 0ff7cb54d38a436e95ebbcd1a66e70aa; no voters:
I20260925 15:08:55.437317 8647 raft_consensus.cc:2781] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:08:55.437443 8647 raft_consensus.cc:482] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:08:55.437541 8647 raft_consensus.cc:3037] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:55.439617 8647 raft_consensus.cc:504] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35807 } } peers { permanent_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41021 } } peers { permanent_uuid: "986f58304b8b4cb19e79246002a38148" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38903 } }
I20260925 15:08:55.440237 8647 leader_election.cc:288] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [CANDIDATE]: Term 1 election: Requested vote from peers 0ff7cb54d38a436e95ebbcd1a66e70aa (127.22.236.1:35807), 986f58304b8b4cb19e79246002a38148 (127.22.236.3:38903)
I20260925 15:08:55.440691 8311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be78410bbd5146b49013cf2c9ba12626" candidate_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa"
I20260925 15:08:55.440891 8311 raft_consensus.cc:3037] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:55.440869 8578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be78410bbd5146b49013cf2c9ba12626" candidate_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "986f58304b8b4cb19e79246002a38148"
I20260925 15:08:55.441072 8578 raft_consensus.cc:3037] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:55.443137 8311 raft_consensus.cc:2454] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b74e71ee6dd48708fdd7e03da933a9b in term 1.
I20260925 15:08:55.443495 8380 leader_election.cc:302] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b74e71ee6dd48708fdd7e03da933a9b, 0ff7cb54d38a436e95ebbcd1a66e70aa; no voters:
I20260925 15:08:55.443732 8647 raft_consensus.cc:2781] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:08:55.444121 8578 raft_consensus.cc:2454] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b74e71ee6dd48708fdd7e03da933a9b in term 1.
I20260925 15:08:55.444211 8647 raft_consensus.cc:686] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [term 1 LEADER]: Becoming Leader. State: Replica: 0b74e71ee6dd48708fdd7e03da933a9b, State: Running, Role: LEADER
I20260925 15:08:55.444777 8647 consensus_queue.cc:237] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35807 } } peers { permanent_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41021 } } peers { permanent_uuid: "986f58304b8b4cb19e79246002a38148" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38903 } }
I20260925 15:08:55.448522 8166 catalog_manager.cc:5495] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b74e71ee6dd48708fdd7e03da933a9b (127.22.236.2). New cstate: current_term: 1 leader_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b74e71ee6dd48708fdd7e03da933a9b" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "986f58304b8b4cb19e79246002a38148" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38903 } health_report { overall_health: UNKNOWN } } }
I20260925 15:08:55.501947 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.496001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58538 (local address 127.22.236.2:41021)
0925 15:08:55.496203 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:55.496208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:55.496227 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:08:55.496411 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:55.496416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:55.496514 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:55.496655 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:08:55.496938 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.498090 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.499519 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.500209 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.500312 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.500773 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:08:55.501397 (+ 624us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:08:55.501542 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:55.501590 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 15:08:55.501665 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:08:55.509888 8578 raft_consensus.cc:1261] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148 [term 1 FOLLOWER]: Refusing update from remote peer 0b74e71ee6dd48708fdd7e03da933a9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:08:55.509899 8311 raft_consensus.cc:1261] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa [term 1 FOLLOWER]: Refusing update from remote peer 0b74e71ee6dd48708fdd7e03da933a9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:08:55.510746 8652 consensus_queue.cc:1035] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ff7cb54d38a436e95ebbcd1a66e70aa" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:08:55.511113 8647 consensus_queue.cc:1035] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "986f58304b8b4cb19e79246002a38148" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:08:55.521159 8655 mvcc.cc:205] Tried to move back new op lower bound from 7333269239838420992 to 7333269239585308672. Current Snapshot: MvccSnapshot[applied={T|T < 7333269239838420992}]
I20260925 15:08:55.535668 8166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:08:55.535940 8166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:08:55.539172 8166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ed5954303a24ac5a910ef926d24cf06: Sending DeleteTablet for 3 replicas of tablet be78410bbd5146b49013cf2c9ba12626
I20260925 15:08:55.540307 8425 tablet_service.cc:1514] Processing DeleteTablet for tablet be78410bbd5146b49013cf2c9ba12626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:08:55 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:58534
I20260925 15:08:55.540362 8291 tablet_service.cc:1514] Processing DeleteTablet for tablet be78410bbd5146b49013cf2c9ba12626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:08:55 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:53514
I20260925 15:08:55.540503 8558 tablet_service.cc:1514] Processing DeleteTablet for tablet be78410bbd5146b49013cf2c9ba12626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:08:55 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:33798
I20260925 15:08:55.542551 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 8228
I20260925 15:08:55.551644 8664 tablet_replica.cc:330] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b: stopping tablet replica
I20260925 15:08:55.552178 8664 raft_consensus.cc:2229] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:08:55.552891 8664 raft_consensus.cc:2258] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:08:55.553755 8666 tablet_replica.cc:330] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148: stopping tablet replica
I20260925 15:08:55.554004 8665 tablet_replica.cc:330] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa: stopping tablet replica
I20260925 15:08:55.554186 8666 raft_consensus.cc:2229] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:08:55.554332 8665 raft_consensus.cc:2229] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:08:55.554683 8666 raft_consensus.cc:2258] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:08:55.554760 8665 raft_consensus.cc:2258] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:08:55.557082 8665 ts_tablet_manager.cc:1905] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:08:55.557194 8664 ts_tablet_manager.cc:1905] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:08:55.558454 8666 ts_tablet_manager.cc:1905] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:08:55.564043 8665 ts_tablet_manager.cc:1918] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:08:55.564299 8665 log.cc:1196] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/be78410bbd5146b49013cf2c9ba12626
I20260925 15:08:55.564925 8665 ts_tablet_manager.cc:1939] T be78410bbd5146b49013cf2c9ba12626 P 0ff7cb54d38a436e95ebbcd1a66e70aa: Deleting consensus metadata
I20260925 15:08:55.565532 8666 ts_tablet_manager.cc:1918] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:08:55.565747 8666 log.cc:1196] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/be78410bbd5146b49013cf2c9ba12626
I20260925 15:08:55.566274 8666 ts_tablet_manager.cc:1939] T be78410bbd5146b49013cf2c9ba12626 P 986f58304b8b4cb19e79246002a38148: Deleting consensus metadata
I20260925 15:08:55.566874 8153 catalog_manager.cc:4841] TS 0ff7cb54d38a436e95ebbcd1a66e70aa (127.22.236.1:35807): tablet be78410bbd5146b49013cf2c9ba12626 (table test-table [id=4509f05e182c4c428ee66330b0676081]) successfully deleted
I20260925 15:08:55.567953 8153 catalog_manager.cc:4841] TS 986f58304b8b4cb19e79246002a38148 (127.22.236.3:38903): tablet be78410bbd5146b49013cf2c9ba12626 (table test-table [id=4509f05e182c4c428ee66330b0676081]) successfully deleted
I20260925 15:08:55.570355 8664 ts_tablet_manager.cc:1918] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:08:55.570538 8664 log.cc:1196] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/be78410bbd5146b49013cf2c9ba12626
Sep 25 15:08:55 dist-test-slave-nsrc krb5kdc[8122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:08:55.571064 8664 ts_tablet_manager.cc:1939] T be78410bbd5146b49013cf2c9ba12626 P 0b74e71ee6dd48708fdd7e03da933a9b: Deleting consensus metadata
I20260925 15:08:55.572741 8152 catalog_manager.cc:4841] TS 0b74e71ee6dd48708fdd7e03da933a9b (127.22.236.2:41021): tablet be78410bbd5146b49013cf2c9ba12626 (table test-table [id=4509f05e182c4c428ee66330b0676081]) successfully deleted
I20260925 15:08:55.578838 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.543734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.22.236.1:35807)
0925 15:08:55.543944 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:55.543952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:55.557996 (+ 14044us) server_negotiation.cc:407] Connection header received
0925 15:08:55.558269 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:55.558275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:55.558374 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:55.558458 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:55.560208 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.561259 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.563068 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.563704 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.563794 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.572288 (+ 8494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:55.572304 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:55.572322 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:55.572371 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:55.575854 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:55.576544 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:55.576554 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:55.576557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:55.576746 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:55.577138 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:55.577143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:55.577147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:55.577585 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:55.577882 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:55.578239 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 15:08:55.578455 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:08:55.579547 8351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:55.693292 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 8228
I20260925 15:08:55.707300 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 8362
Sep 25 15:08:55 dist-test-slave-nsrc krb5kdc[8122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:08:55.722168 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.708122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.22.236.2:41021)
0925 15:08:55.708326 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:55.708332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:55.708353 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:08:55.708416 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:55.708420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:55.708481 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:55.708564 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:55.709766 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.710603 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.711918 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.712371 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.712484 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.716689 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:55.716704 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:55.716719 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:55.716757 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:55.719576 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:55.720222 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:55.720231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:55.720233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:55.720367 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:55.720713 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:55.720718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:55.720720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:55.721072 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:55.721408 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:55.721606 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 15:08:55.721802 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:08:55.722867 8485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:55.834481 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 8362
I20260925 15:08:55.848786 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 8495
Sep 25 15:08:55 dist-test-slave-nsrc krb5kdc[8122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:08:55.863126 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:55.849504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.22.236.3:38903)
0925 15:08:55.849656 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:55.849661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:55.849678 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:08:55.849813 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:55.849817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:55.849881 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:55.849969 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:55.851249 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.851955 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.853290 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:55.853669 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:55.853758 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:55.857837 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:55.857851 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:55.857867 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:55.857898 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:55.860502 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:55.861122 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:55.861130 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:55.861132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:55.861266 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:55.861622 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:55.861626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:55.861628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:55.862006 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:55.862270 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:55.862592 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 15:08:55.862801 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:08:55.863780 8618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:55.949774 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 8495
I20260925 15:08:55.962889 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 8135
I20260925 15:08:55.963960 8197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:56.072480 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 8135
2026-09-25T15:08:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4288 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T15:08:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:08:56Z Disabled control of system clock
I20260925 15:08:56.141639 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:43879
--webserver_interface=127.22.236.62
--webserver_port=0
--builtin_ntp_servers=127.22.236.20:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:43879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:08:56.456084 8675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:56.456384 8675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:56.456467 8675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:56.463631 8675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:08:56.463744 8675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:56.463789 8675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:56.463848 8675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:08:56.463891 8675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:08:56.473240 8675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:43879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:43879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:56.474538 8675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:56.476277 8675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:56.487056 8681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:56.487011 8680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:56.487597 8683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:56.488227 8675 server_base.cc:1034] running on GCE node
I20260925 15:08:56.488950 8675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:56.490280 8675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:56.491473 8675 hybrid_clock.cc:648] HybridClock initialized: now 1790348936491422 us; error 45 us; skew 500 ppm
I20260925 15:08:56.491779 8675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:08:56.497692 8675 webserver.cc:460] Webserver started at http://127.22.236.62:36623/ using document root <none> and password file <none>
I20260925 15:08:56.498337 8675 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:56.498433 8675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:56.498724 8675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:56.501601 8675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "66a964c9101b4d01a878c7e2e018ebd5"
format_stamp: "Formatted at 2026-09-25 15:08:56 on dist-test-slave-nsrc"
server_key: "45688c64dd5410c44fc397fc81a7f749"
server_key_iv: "147cee12c7f9a28557e7896d79fe6c7f"
server_key_version: "encryptionkey@0"
I20260925 15:08:56.502348 8675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "66a964c9101b4d01a878c7e2e018ebd5"
format_stamp: "Formatted at 2026-09-25 15:08:56 on dist-test-slave-nsrc"
server_key: "45688c64dd5410c44fc397fc81a7f749"
server_key_iv: "147cee12c7f9a28557e7896d79fe6c7f"
server_key_version: "encryptionkey@0"
I20260925 15:08:56.507121 8675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:08:56.510377 8689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:56.511703 8675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:08:56.511869 8675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "66a964c9101b4d01a878c7e2e018ebd5"
format_stamp: "Formatted at 2026-09-25 15:08:56 on dist-test-slave-nsrc"
server_key: "45688c64dd5410c44fc397fc81a7f749"
server_key_iv: "147cee12c7f9a28557e7896d79fe6c7f"
server_key_version: "encryptionkey@0"
I20260925 15:08:56.512085 8675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:56.526911 8675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:56.527676 8675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:56.527940 8675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:56.544597 8675 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:43879
I20260925 15:08:56.544615 8740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:43879 every 8 connection(s)
I20260925 15:08:56.545907 8675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:08:56.549433 8741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:56.552016 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 8675
I20260925 15:08:56.552196 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:08:56.552631 23472 external_mini_cluster.cc:1428] Setting key 6f42a64ef77e3aee65e9bdd6ab8ddd63
I20260925 15:08:56.561021 8741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5: Bootstrap starting.
I20260925 15:08:56.564584 8741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:56.565716 8741 log.cc:824] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:56.566171 8744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:56.554582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42384 (local address 127.22.236.62:43879)
0925 15:08:56.555523 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:56.555534 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:56.555575 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:08:56.557256 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:56.557282 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:56.557759 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:56.558086 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:56.558663 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:56.559882 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:56.561820 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:56.562478 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:56.562614 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:56.563831 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:56.563838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:56.563852 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:56.564409 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:56.564506 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:56.564527 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:56.564623 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:56.564915 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:08:56.565216 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":338,"threads_started":1}
I20260925 15:08:56.568573 8741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5: No bootstrap required, opened a new log
I20260925 15:08:56.575564 8741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a964c9101b4d01a878c7e2e018ebd5" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43879 } }
I20260925 15:08:56.575884 8741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:56.575973 8741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a964c9101b4d01a878c7e2e018ebd5, State: Initialized, Role: FOLLOWER
I20260925 15:08:56.576472 8741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a964c9101b4d01a878c7e2e018ebd5" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43879 } }
I20260925 15:08:56.576673 8741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:08:56.576762 8741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:08:56.576879 8741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:56.578855 8741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a964c9101b4d01a878c7e2e018ebd5" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43879 } }
I20260925 15:08:56.579286 8741 leader_election.cc:302] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66a964c9101b4d01a878c7e2e018ebd5; no voters:
I20260925 15:08:56.579792 8741 leader_election.cc:288] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:08:56.579910 8746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:08:56.580552 8746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [term 1 LEADER]: Becoming Leader. State: Replica: 66a964c9101b4d01a878c7e2e018ebd5, State: Running, Role: LEADER
I20260925 15:08:56.581053 8746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a964c9101b4d01a878c7e2e018ebd5" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43879 } }
I20260925 15:08:56.581723 8741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:08:56.583807 8747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66a964c9101b4d01a878c7e2e018ebd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a964c9101b4d01a878c7e2e018ebd5" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43879 } } }
I20260925 15:08:56.584599 8747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:08:56.584321 8748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66a964c9101b4d01a878c7e2e018ebd5. Latest consensus state: current_term: 1 leader_uuid: "66a964c9101b4d01a878c7e2e018ebd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a964c9101b4d01a878c7e2e018ebd5" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 43879 } } }
I20260925 15:08:56.588899 8748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:08:56.589433 8753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:08:56.595170 8753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:08:56.603569 8753 catalog_manager.cc:1347] Generated new cluster ID: f1bf1c44ab7f4071976d50fda4228aed
I20260925 15:08:56.603685 8753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:08:56.631915 8753 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:08:56.633241 8753 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:08:56.644690 8753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5: Generated new TSK 0
I20260925 15:08:56.645673 8753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:08:56.660913 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43879
--builtin_ntp_servers=127.22.236.20:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:08:56.975497 8765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:56.975766 8765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:56.975852 8765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:56.983000 8765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:56.983109 8765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:56.983223 8765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:08:56.992305 8765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:56.993613 8765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:56.995345 8765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:57.008062 8770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:57.008098 8771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:57.008025 8773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:57.009552 8765 server_base.cc:1034] running on GCE node
I20260925 15:08:57.010133 8765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:57.011072 8765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:57.012223 8765 hybrid_clock.cc:648] HybridClock initialized: now 1790348937012169 us; error 43 us; skew 500 ppm
I20260925 15:08:57.012527 8765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:08:57.014715 8765 webserver.cc:460] Webserver started at http://127.22.236.1:36169/ using document root <none> and password file <none>
I20260925 15:08:57.015445 8765 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:57.015543 8765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:57.015828 8765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:57.018729 8765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "6768e5a1508a41399bc97a905cb7599c"
format_stamp: "Formatted at 2026-09-25 15:08:57 on dist-test-slave-nsrc"
server_key: "6589255f5014a954e6334d487e56e756"
server_key_iv: "e2156d4c5484fd8f492a6a41db1b05b0"
server_key_version: "encryptionkey@0"
I20260925 15:08:57.019481 8765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "6768e5a1508a41399bc97a905cb7599c"
format_stamp: "Formatted at 2026-09-25 15:08:57 on dist-test-slave-nsrc"
server_key: "6589255f5014a954e6334d487e56e756"
server_key_iv: "e2156d4c5484fd8f492a6a41db1b05b0"
server_key_version: "encryptionkey@0"
I20260925 15:08:57.024263 8765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 15:08:57.027470 8779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:57.028728 8765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:08:57.028872 8765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "6768e5a1508a41399bc97a905cb7599c"
format_stamp: "Formatted at 2026-09-25 15:08:57 on dist-test-slave-nsrc"
server_key: "6589255f5014a954e6334d487e56e756"
server_key_iv: "e2156d4c5484fd8f492a6a41db1b05b0"
server_key_version: "encryptionkey@0"
I20260925 15:08:57.029078 8765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:57.078033 8765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:57.078858 8765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:57.079151 8765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:57.080097 8765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:57.081570 8765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:57.081665 8765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:57.081766 8765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:57.081832 8765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:57.114619 8765 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:38525
I20260925 15:08:57.114655 8891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:38525 every 8 connection(s)
I20260925 15:08:57.115942 8765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:08:57.122745 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 8765
I20260925 15:08:57.122888 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:08:57.123320 23472 external_mini_cluster.cc:1428] Setting key 4fa30f757a3e837ecc196762547ccd7c
I20260925 15:08:57.125895 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43879
--builtin_ntp_servers=127.22.236.20:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:08:57.128296 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:57.118368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:37757 (local address 127.22.236.62:43879)
0925 15:08:57.118750 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:57.118755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:57.119978 (+ 1223us) server_negotiation.cc:407] Connection header received
0925 15:08:57.121796 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:57.121801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:57.121886 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:57.121982 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:57.123072 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:57.123827 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:57.125634 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:57.126034 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:57.126153 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:57.127593 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:57.127597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:57.127600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:57.127616 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:57.127657 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:57.127668 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:57.127732 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:57.127936 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 15:08:57.128009 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":185,"threads_started":1}
I20260925 15:08:57.129573 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:57.118977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43879 (local address 127.22.236.1:37757)
0925 15:08:57.119823 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:57.119857 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:57.121525 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:57.122219 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:57.122227 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:57.122317 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:57.122899 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:57.122920 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:57.123974 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:57.123979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:57.125498 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:57.125514 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:57.126254 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:57.126261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:57.126555 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:57.127258 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:08:57.127262 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:57.127314 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:08:57.127406 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:57.127750 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:57.127758 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:57.127765 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:08:57.127927 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 15:08:57.128343 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":245,"threads_started":1}
I20260925 15:08:57.131062 8892 heartbeater.cc:347] Connected to a master server at 127.22.236.62:43879
I20260925 15:08:57.131321 8892 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:57.132014 8892 heartbeater.cc:511] Master 127.22.236.62:43879 requested a full tablet report, sending...
I20260925 15:08:57.134116 8706 ts_manager.cc:194] Registered new tserver with Master: 6768e5a1508a41399bc97a905cb7599c (127.22.236.1:38525)
I20260925 15:08:57.135558 8706 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.236.1:37757
W20260925 15:08:57.439143 8896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:57.439441 8896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:57.439570 8896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:57.446924 8896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:57.447031 8896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:57.447165 8896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:08:57.456461 8896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:57.457691 8896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:57.459405 8896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:57.472177 8901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:57.472215 8904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:57.472239 8902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:57.473134 8896 server_base.cc:1034] running on GCE node
I20260925 15:08:57.473762 8896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:57.474701 8896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:57.475895 8896 hybrid_clock.cc:648] HybridClock initialized: now 1790348937475836 us; error 53 us; skew 500 ppm
I20260925 15:08:57.476233 8896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:08:57.478423 8896 webserver.cc:460] Webserver started at http://127.22.236.2:36319/ using document root <none> and password file <none>
I20260925 15:08:57.479092 8896 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:57.479190 8896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:57.479483 8896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:57.482537 8896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "17682c7576b54f92bbc1c998fe9c0a29"
format_stamp: "Formatted at 2026-09-25 15:08:57 on dist-test-slave-nsrc"
server_key: "45f2bedee2c5fabba6dc3c2944667aac"
server_key_iv: "d520395225bc7121f62183ccc495807a"
server_key_version: "encryptionkey@0"
I20260925 15:08:57.483330 8896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "17682c7576b54f92bbc1c998fe9c0a29"
format_stamp: "Formatted at 2026-09-25 15:08:57 on dist-test-slave-nsrc"
server_key: "45f2bedee2c5fabba6dc3c2944667aac"
server_key_iv: "d520395225bc7121f62183ccc495807a"
server_key_version: "encryptionkey@0"
I20260925 15:08:57.488296 8896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 15:08:57.491657 8910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:57.492982 8896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:08:57.493132 8896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "17682c7576b54f92bbc1c998fe9c0a29"
format_stamp: "Formatted at 2026-09-25 15:08:57 on dist-test-slave-nsrc"
server_key: "45f2bedee2c5fabba6dc3c2944667aac"
server_key_iv: "d520395225bc7121f62183ccc495807a"
server_key_version: "encryptionkey@0"
I20260925 15:08:57.493320 8896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:57.506903 8896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:57.507696 8896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:57.507970 8896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:57.509071 8896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:57.510490 8896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:57.510584 8896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:57.510695 8896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:57.510766 8896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:57.539351 8896 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:44357
I20260925 15:08:57.539405 9022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:44357 every 8 connection(s)
I20260925 15:08:57.540683 8896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:08:57.549118 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 8896
I20260925 15:08:57.549270 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:08:57.549839 23472 external_mini_cluster.cc:1428] Setting key 6fd894f4c8efd0918cf616036e4c5086
I20260925 15:08:57.552780 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:57.543084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:37723 (local address 127.22.236.62:43879)
0925 15:08:57.543253 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:57.543257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:57.544691 (+ 1434us) server_negotiation.cc:407] Connection header received
0925 15:08:57.546404 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:57.546410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:57.546485 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:57.546570 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:57.547581 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:57.548386 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:57.550013 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:57.550459 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:57.550582 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:57.551981 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:57.551987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:57.551992 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:57.552011 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:57.552052 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:57.552067 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:57.552175 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:57.552384 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 15:08:57.552479 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:08:57.552937 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43879
--builtin_ntp_servers=127.22.236.20:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:08:57.553958 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:57.543605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43879 (local address 127.22.236.2:37723)
0925 15:08:57.544545 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:57.544579 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:57.546148 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:57.546795 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:57.546803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:57.546888 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:57.547422 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:57.547440 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:57.548532 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:57.548537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:57.549819 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:57.549831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:57.550700 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:57.550707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:57.551011 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:57.551658 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:08:57.551662 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:57.551708 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:08:57.551794 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:57.552187 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:57.552196 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:57.552203 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:08:57.552360 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 15:08:57.552732 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":296,"threads_started":1}
I20260925 15:08:57.555387 9023 heartbeater.cc:347] Connected to a master server at 127.22.236.62:43879
I20260925 15:08:57.555675 9023 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:57.556577 9023 heartbeater.cc:511] Master 127.22.236.62:43879 requested a full tablet report, sending...
I20260925 15:08:57.558353 8706 ts_manager.cc:194] Registered new tserver with Master: 17682c7576b54f92bbc1c998fe9c0a29 (127.22.236.2:44357)
I20260925 15:08:57.559340 8706 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.236.2:37723
W20260925 15:08:57.861969 9026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:57.862232 9026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:57.862361 9026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:57.869513 9026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:57.869619 9026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:57.869765 9026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:08:57.879020 9026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:08:57.880268 9026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:57.882042 9026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:57.894397 9031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:57.894446 9032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:57.894465 9034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:57.895610 9026 server_base.cc:1034] running on GCE node
I20260925 15:08:57.896245 9026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:57.897272 9026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:57.898468 9026 hybrid_clock.cc:648] HybridClock initialized: now 1790348937898418 us; error 45 us; skew 500 ppm
I20260925 15:08:57.898757 9026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:08:57.900954 9026 webserver.cc:460] Webserver started at http://127.22.236.3:36891/ using document root <none> and password file <none>
I20260925 15:08:57.901603 9026 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:57.901701 9026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:57.901990 9026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:57.904806 9026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "61e238aa7fa84f0d85a3629d03473b81"
format_stamp: "Formatted at 2026-09-25 15:08:57 on dist-test-slave-nsrc"
server_key: "726d4356b081f2633cec4b5987b4dac7"
server_key_iv: "f20da2958c134eb0eb608b8d7cf4fec8"
server_key_version: "encryptionkey@0"
I20260925 15:08:57.905602 9026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "61e238aa7fa84f0d85a3629d03473b81"
format_stamp: "Formatted at 2026-09-25 15:08:57 on dist-test-slave-nsrc"
server_key: "726d4356b081f2633cec4b5987b4dac7"
server_key_iv: "f20da2958c134eb0eb608b8d7cf4fec8"
server_key_version: "encryptionkey@0"
I20260925 15:08:57.910353 9026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 15:08:57.913630 9040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:57.914872 9026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:08:57.915020 9026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "61e238aa7fa84f0d85a3629d03473b81"
format_stamp: "Formatted at 2026-09-25 15:08:57 on dist-test-slave-nsrc"
server_key: "726d4356b081f2633cec4b5987b4dac7"
server_key_iv: "f20da2958c134eb0eb608b8d7cf4fec8"
server_key_version: "encryptionkey@0"
I20260925 15:08:57.915203 9026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:57.933300 9026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:57.934098 9026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:57.934370 9026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:57.935299 9026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:57.936690 9026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:57.936785 9026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:57.936887 9026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:57.936962 9026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:57.964803 9026 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:35773
I20260925 15:08:57.964857 9152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:35773 every 8 connection(s)
I20260925 15:08:57.966045 9026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:08:57.974331 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 9026
I20260925 15:08:57.974490 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:08:57.974896 23472 external_mini_cluster.cc:1428] Setting key 5847697c9aabd84916c66173ad9ef0ed
I20260925 15:08:57.978147 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:57.968405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:46157 (local address 127.22.236.62:43879)
0925 15:08:57.968562 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:57.968566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:57.969959 (+ 1393us) server_negotiation.cc:407] Connection header received
0925 15:08:57.971661 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:57.971667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:57.971747 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:57.971847 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:57.972939 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:57.973711 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:57.975324 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:57.975852 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:57.975976 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:57.977439 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:57.977447 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:57.977451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:57.977472 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:57.977510 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:57.977527 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:57.977594 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:57.977840 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 15:08:57.977919 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:08:57.979076 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:57.968920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:43879 (local address 127.22.236.3:46157)
0925 15:08:57.969790 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:57.969831 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:57.971397 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:57.972082 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:57.972091 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:57.972213 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:57.972771 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:57.972789 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:57.973867 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:57.973870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:57.975189 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:57.975202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:57.976083 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:57.976090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:57.976341 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:57.977055 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:08:57.977059 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:57.977111 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:08:57.977222 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:57.977620 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:57.977631 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:57.977640 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:08:57.977791 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 15:08:57.978087 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":286,"threads_started":1}
I20260925 15:08:57.980265 9153 heartbeater.cc:347] Connected to a master server at 127.22.236.62:43879
I20260925 15:08:57.980535 9153 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:57.981248 9153 heartbeater.cc:511] Master 127.22.236.62:43879 requested a full tablet report, sending...
I20260925 15:08:57.982538 8706 ts_manager.cc:194] Registered new tserver with Master: 61e238aa7fa84f0d85a3629d03473b81 (127.22.236.3:35773)
I20260925 15:08:57.983343 8706 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.236.3:46157
I20260925 15:08:57.989511 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:08:57.998835 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:57.993389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36850 (local address 127.22.236.62:43879)
0925 15:08:57.993563 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:57.993567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:57.994030 (+ 463us) server_negotiation.cc:407] Connection header received
0925 15:08:57.994182 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:57.994186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:57.994257 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:57.994330 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:57.994737 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:57.995387 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:57.996861 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:57.997321 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:57.997438 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:57.998309 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:57.998314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:57.998317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:57.998333 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:57.998364 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:57.998375 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:57.998445 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:57.998529 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 15:08:57.998606 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:08:58.002257 8706 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:08:58.008857 8706 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:08:58.030987 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.020214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38525 (local address 127.0.0.1:38626)
0925 15:08:58.021905 (+ 1691us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:58.021938 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:58.022115 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:58.023485 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:58.023493 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:58.023523 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:58.023908 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:58.023920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.025771 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.025777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:58.028030 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:58.028049 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.030116 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.030123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:58.030301 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:58.030351 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:08:58.030493 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 15:08:58.030666 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1486,"thread_start_us":1350,"threads_started":1}
I20260925 15:08:58.033238 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.020416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.22.236.1:38525)
0925 15:08:58.020931 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:58.020939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:58.022051 (+ 1112us) server_negotiation.cc:407] Connection header received
0925 15:08:58.022311 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:58.022320 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:58.023029 (+ 709us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:58.023202 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:58.024100 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.025639 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.028290 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.029638 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.029763 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:58.029982 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:58.031739 (+ 1757us) server_negotiation.cc:299] Negotiation successful
0925 15:08:58.031912 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":572,"threads_started":1}
I20260925 15:08:58.034323 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.020519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:44357 (local address 127.0.0.1:34574)
0925 15:08:58.022627 (+ 2108us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:58.022643 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:58.023026 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:58.023867 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:58.023871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:58.023907 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:58.024303 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:58.024316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.027129 (+ 2813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.027134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:58.030116 (+ 2982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:58.030127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.032179 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.032185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:58.032733 (+ 548us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:58.032765 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:08:58.032852 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:08:58.032981 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1980,"thread_start_us":1838,"threads_started":1}
I20260925 15:08:58.035262 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.020835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.22.236.2:44357)
0925 15:08:58.021354 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:58.021362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:58.022947 (+ 1585us) server_negotiation.cc:407] Connection header received
0925 15:08:58.023223 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:58.023230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:58.023433 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:58.023613 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:58.024500 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.026967 (+ 2467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.030313 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.031647 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.031766 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:58.032030 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:58.034732 (+ 2702us) server_negotiation.cc:299] Negotiation successful
0925 15:08:58.034908 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":477,"threads_started":1}
I20260925 15:08:58.037261 8827 tablet_service.cc:1467] Processing CreateTablet for tablet 9af227cf49c3401c8f2289456bd8dccd (DEFAULT_TABLE table=test-table [id=76c9e9b61e4242cc843d91872787ace6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:08:58.038767 8827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af227cf49c3401c8f2289456bd8dccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:58.039139 8958 tablet_service.cc:1467] Processing CreateTablet for tablet 9af227cf49c3401c8f2289456bd8dccd (DEFAULT_TABLE table=test-table [id=76c9e9b61e4242cc843d91872787ace6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:08:58.040496 8958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af227cf49c3401c8f2289456bd8dccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:58.043361 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.022679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:35773 (local address 127.0.0.1:58508)
0925 15:08:58.031470 (+ 8791us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:58.031488 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:58.031575 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:58.037010 (+ 5435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:58.037016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:58.037046 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:58.037207 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:58.037218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.039226 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.039232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:58.041007 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:58.041022 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.042635 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.042641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:58.042818 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:58.042852 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:08:58.042942 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:08:58.043074 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8671,"thread_start_us":13159,"threads_started":1}
I20260925 15:08:58.044545 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.023026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.22.236.3:35773)
0925 15:08:58.036458 (+ 13432us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:58.036466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:58.036503 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:08:58.036593 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:58.036601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:58.036825 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:58.037019 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:58.037590 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.039082 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.041169 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.042468 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.042601 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:58.042848 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:58.044004 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0925 15:08:58.044197 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13260,"thread_start_us":13108,"threads_started":1}
I20260925 15:08:58.048180 9088 tablet_service.cc:1467] Processing CreateTablet for tablet 9af227cf49c3401c8f2289456bd8dccd (DEFAULT_TABLE table=test-table [id=76c9e9b61e4242cc843d91872787ace6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:08:58.049626 9088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af227cf49c3401c8f2289456bd8dccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:58.052096 9171 tablet_bootstrap.cc:492] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c: Bootstrap starting.
I20260925 15:08:58.055220 9171 tablet_bootstrap.cc:654] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:58.056226 9171 log.cc:824] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:58.056849 9172 tablet_bootstrap.cc:492] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29: Bootstrap starting.
I20260925 15:08:58.059990 9172 tablet_bootstrap.cc:654] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:58.060132 9171 tablet_bootstrap.cc:492] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c: No bootstrap required, opened a new log
I20260925 15:08:58.060451 9171 ts_tablet_manager.cc:1397] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 15:08:58.061275 9172 log.cc:824] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:58.064073 9172 tablet_bootstrap.cc:492] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29: No bootstrap required, opened a new log
I20260925 15:08:58.064355 9172 ts_tablet_manager.cc:1397] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:08:58.064874 9175 tablet_bootstrap.cc:492] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81: Bootstrap starting.
I20260925 15:08:58.067662 9175 tablet_bootstrap.cc:654] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:58.068598 9175 log.cc:824] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:58.070092 9171 raft_consensus.cc:348] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6768e5a1508a41399bc97a905cb7599c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38525 } } peers { permanent_uuid: "17682c7576b54f92bbc1c998fe9c0a29" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44357 } } peers { permanent_uuid: "61e238aa7fa84f0d85a3629d03473b81" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35773 } }
I20260925 15:08:58.070616 9171 raft_consensus.cc:374] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:58.070749 9171 raft_consensus.cc:729] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6768e5a1508a41399bc97a905cb7599c, State: Initialized, Role: FOLLOWER
I20260925 15:08:58.071280 9175 tablet_bootstrap.cc:492] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81: No bootstrap required, opened a new log
I20260925 15:08:58.071518 9175 ts_tablet_manager.cc:1397] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 15:08:58.071388 9171 consensus_queue.cc:260] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6768e5a1508a41399bc97a905cb7599c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38525 } } peers { permanent_uuid: "17682c7576b54f92bbc1c998fe9c0a29" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44357 } } peers { permanent_uuid: "61e238aa7fa84f0d85a3629d03473b81" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35773 } }
I20260925 15:08:58.072511 8892 heartbeater.cc:503] Master 127.22.236.62:43879 was elected leader, sending a full tablet report...
I20260925 15:08:58.073099 9171 ts_tablet_manager.cc:1428] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260925 15:08:58.074033 9172 raft_consensus.cc:348] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6768e5a1508a41399bc97a905cb7599c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38525 } } peers { permanent_uuid: "17682c7576b54f92bbc1c998fe9c0a29" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44357 } } peers { permanent_uuid: "61e238aa7fa84f0d85a3629d03473b81" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35773 } }
I20260925 15:08:58.074445 9172 raft_consensus.cc:374] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:58.074549 9172 raft_consensus.cc:729] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17682c7576b54f92bbc1c998fe9c0a29, State: Initialized, Role: FOLLOWER
I20260925 15:08:58.075084 9172 consensus_queue.cc:260] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6768e5a1508a41399bc97a905cb7599c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38525 } } peers { permanent_uuid: "17682c7576b54f92bbc1c998fe9c0a29" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44357 } } peers { permanent_uuid: "61e238aa7fa84f0d85a3629d03473b81" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35773 } }
I20260925 15:08:58.076116 9023 heartbeater.cc:503] Master 127.22.236.62:43879 was elected leader, sending a full tablet report...
I20260925 15:08:58.076766 9172 ts_tablet_manager.cc:1428] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260925 15:08:58.080477 9175 raft_consensus.cc:348] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6768e5a1508a41399bc97a905cb7599c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38525 } } peers { permanent_uuid: "17682c7576b54f92bbc1c998fe9c0a29" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44357 } } peers { permanent_uuid: "61e238aa7fa84f0d85a3629d03473b81" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35773 } }
I20260925 15:08:58.080773 9175 raft_consensus.cc:374] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:58.080837 9175 raft_consensus.cc:729] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e238aa7fa84f0d85a3629d03473b81, State: Initialized, Role: FOLLOWER
I20260925 15:08:58.081236 9175 consensus_queue.cc:260] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6768e5a1508a41399bc97a905cb7599c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38525 } } peers { permanent_uuid: "17682c7576b54f92bbc1c998fe9c0a29" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44357 } } peers { permanent_uuid: "61e238aa7fa84f0d85a3629d03473b81" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35773 } }
I20260925 15:08:58.082036 9153 heartbeater.cc:503] Master 127.22.236.62:43879 was elected leader, sending a full tablet report...
I20260925 15:08:58.082623 9175 ts_tablet_manager.cc:1428] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
W20260925 15:08:58.118176 8893 tablet.cc:2369] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:08:58.145395 9177 raft_consensus.cc:482] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:08:58.145658 9177 raft_consensus.cc:504] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6768e5a1508a41399bc97a905cb7599c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38525 } } peers { permanent_uuid: "17682c7576b54f92bbc1c998fe9c0a29" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44357 } } peers { permanent_uuid: "61e238aa7fa84f0d85a3629d03473b81" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35773 } }
I20260925 15:08:58.147256 9177 leader_election.cc:288] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17682c7576b54f92bbc1c998fe9c0a29 (127.22.236.2:44357), 61e238aa7fa84f0d85a3629d03473b81 (127.22.236.3:35773)
I20260925 15:08:58.153358 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.147669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:49669 (local address 127.22.236.2:44357)
0925 15:08:58.147837 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:58.147841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:58.148163 (+ 322us) server_negotiation.cc:407] Connection header received
0925 15:08:58.148451 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:58.148457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:58.148538 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:58.148602 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:58.149198 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.149879 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.151766 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.152538 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.152684 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:58.152754 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:58.153002 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 15:08:58.153123 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:08:58.153445 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.147445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:44357 (local address 127.22.236.1:49669)
0925 15:08:58.148052 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:58.148070 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:58.148235 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:58.148771 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:58.148774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:58.148795 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:58.149064 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:58.149071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.150027 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.150031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:58.151597 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:58.151609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.152708 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.152714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:58.152858 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:58.152881 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:08:58.152963 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:08:58.153090 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":92,"thread_start_us":289,"threads_started":1}
I20260925 15:08:58.153553 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.148273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:45203 (local address 127.22.236.3:35773)
0925 15:08:58.148457 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:58.148463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:58.148751 (+ 288us) server_negotiation.cc:407] Connection header received
0925 15:08:58.148917 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:58.148921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:58.148976 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:58.149045 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:58.149533 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.150233 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.151957 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.152740 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.152903 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:58.152970 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:58.153226 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 15:08:58.153339 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:08:58.153851 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.148125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:35773 (local address 127.22.236.1:45203)
0925 15:08:58.148658 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:58.148671 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:58.148789 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:58.149080 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:58.149084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:58.149105 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:58.149399 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:58.149408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.150355 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.150358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:58.151792 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:58.151804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.152925 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.152933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:58.153095 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:58.153117 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:08:58.153191 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:08:58.153306 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":40,"thread_start_us":269,"threads_started":1}
I20260925 15:08:58.154561 8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af227cf49c3401c8f2289456bd8dccd" candidate_uuid: "6768e5a1508a41399bc97a905cb7599c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17682c7576b54f92bbc1c998fe9c0a29" is_pre_election: true
I20260925 15:08:58.154704 9108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af227cf49c3401c8f2289456bd8dccd" candidate_uuid: "6768e5a1508a41399bc97a905cb7599c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61e238aa7fa84f0d85a3629d03473b81" is_pre_election: true
I20260925 15:08:58.155026 8978 raft_consensus.cc:2454] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6768e5a1508a41399bc97a905cb7599c in term 0.
I20260925 15:08:58.155162 9108 raft_consensus.cc:2454] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6768e5a1508a41399bc97a905cb7599c in term 0.
I20260925 15:08:58.155614 8783 leader_election.cc:302] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17682c7576b54f92bbc1c998fe9c0a29, 6768e5a1508a41399bc97a905cb7599c; no voters:
I20260925 15:08:58.155939 9177 raft_consensus.cc:2781] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:08:58.156067 9177 raft_consensus.cc:482] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:08:58.156189 9177 raft_consensus.cc:3037] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:58.158224 9177 raft_consensus.cc:504] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6768e5a1508a41399bc97a905cb7599c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38525 } } peers { permanent_uuid: "17682c7576b54f92bbc1c998fe9c0a29" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44357 } } peers { permanent_uuid: "61e238aa7fa84f0d85a3629d03473b81" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35773 } }
I20260925 15:08:58.158761 9177 leader_election.cc:288] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [CANDIDATE]: Term 1 election: Requested vote from peers 17682c7576b54f92bbc1c998fe9c0a29 (127.22.236.2:44357), 61e238aa7fa84f0d85a3629d03473b81 (127.22.236.3:35773)
I20260925 15:08:58.159192 8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af227cf49c3401c8f2289456bd8dccd" candidate_uuid: "6768e5a1508a41399bc97a905cb7599c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17682c7576b54f92bbc1c998fe9c0a29"
I20260925 15:08:58.159209 9108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af227cf49c3401c8f2289456bd8dccd" candidate_uuid: "6768e5a1508a41399bc97a905cb7599c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61e238aa7fa84f0d85a3629d03473b81"
I20260925 15:08:58.159363 8978 raft_consensus.cc:3037] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:58.159361 9108 raft_consensus.cc:3037] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:58.161589 9108 raft_consensus.cc:2454] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6768e5a1508a41399bc97a905cb7599c in term 1.
I20260925 15:08:58.161598 8978 raft_consensus.cc:2454] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6768e5a1508a41399bc97a905cb7599c in term 1.
I20260925 15:08:58.161959 8783 leader_election.cc:302] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e238aa7fa84f0d85a3629d03473b81, 6768e5a1508a41399bc97a905cb7599c; no voters:
I20260925 15:08:58.162225 9177 raft_consensus.cc:2781] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:08:58.162642 9177 raft_consensus.cc:686] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [term 1 LEADER]: Becoming Leader. State: Replica: 6768e5a1508a41399bc97a905cb7599c, State: Running, Role: LEADER
I20260925 15:08:58.163161 9177 consensus_queue.cc:237] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6768e5a1508a41399bc97a905cb7599c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38525 } } peers { permanent_uuid: "17682c7576b54f92bbc1c998fe9c0a29" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44357 } } peers { permanent_uuid: "61e238aa7fa84f0d85a3629d03473b81" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35773 } }
I20260925 15:08:58.167750 8706 catalog_manager.cc:5495] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6768e5a1508a41399bc97a905cb7599c (127.22.236.1). New cstate: current_term: 1 leader_uuid: "6768e5a1508a41399bc97a905cb7599c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6768e5a1508a41399bc97a905cb7599c" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17682c7576b54f92bbc1c998fe9c0a29" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e238aa7fa84f0d85a3629d03473b81" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35773 } health_report { overall_health: UNKNOWN } } }
I20260925 15:08:58.183401 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.178586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38630 (local address 127.22.236.1:38525)
0925 15:08:58.178766 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:58.178770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:58.178788 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:08:58.178937 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:58.178941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:58.179032 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:58.179148 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:08:58.179417 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.180102 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.181482 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.181870 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.181963 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:58.182378 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:08:58.182915 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:08:58.183013 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:58.183101 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 15:08:58.183159 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:08:58.191047 9108 raft_consensus.cc:1261] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81 [term 1 FOLLOWER]: Refusing update from remote peer 6768e5a1508a41399bc97a905cb7599c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:08:58.191046 8978 raft_consensus.cc:1261] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29 [term 1 FOLLOWER]: Refusing update from remote peer 6768e5a1508a41399bc97a905cb7599c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:08:58.191938 9182 consensus_queue.cc:1035] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e238aa7fa84f0d85a3629d03473b81" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 35773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:08:58.192308 9177 consensus_queue.cc:1035] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17682c7576b54f92bbc1c998fe9c0a29" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 44357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:08:58.210394 9186 mvcc.cc:205] Tried to move back new op lower bound from 7333269250821320704 to 7333269250720124928. Current Snapshot: MvccSnapshot[applied={T|T < 7333269250821320704}]
I20260925 15:08:58.221060 8706 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:08:58.221333 8706 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:08:58.224875 8706 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66a964c9101b4d01a878c7e2e018ebd5: Sending DeleteTablet for 3 replicas of tablet 9af227cf49c3401c8f2289456bd8dccd
I20260925 15:08:58.225932 8827 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af227cf49c3401c8f2289456bd8dccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:08:58 UTC) from {username='slave'} at 127.0.0.1:38626
I20260925 15:08:58.225975 8958 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af227cf49c3401c8f2289456bd8dccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:08:58 UTC) from {username='slave'} at 127.0.0.1:34574
I20260925 15:08:58.226866 9088 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af227cf49c3401c8f2289456bd8dccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:08:58 UTC) from {username='slave'} at 127.0.0.1:58508
I20260925 15:08:58.227145 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 8765
I20260925 15:08:58.236673 9195 tablet_replica.cc:330] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c: stopping tablet replica
I20260925 15:08:58.237147 9195 raft_consensus.cc:2229] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:08:58.237874 9195 raft_consensus.cc:2258] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:08:58.238622 9196 tablet_replica.cc:330] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29: stopping tablet replica
I20260925 15:08:58.238915 9197 tablet_replica.cc:330] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81: stopping tablet replica
I20260925 15:08:58.239089 9196 raft_consensus.cc:2229] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:08:58.239320 9197 raft_consensus.cc:2229] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:08:58.239626 9196 raft_consensus.cc:2258] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:08:58.239769 9197 raft_consensus.cc:2258] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:08:58.240654 9195 ts_tablet_manager.cc:1905] T 9af227cf49c3401c8f2289456bd8dccd P 6768e5a1508a41399bc97a905cb7599c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:08:58.241570 9196 ts_tablet_manager.cc:1905] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:08:58.241570 9197 ts_tablet_manager.cc:1905] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:08:58.245843 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.228412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.22.236.1:38525)
0925 15:08:58.228600 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:58.228606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:58.238135 (+ 9529us) server_negotiation.cc:407] Connection header received
0925 15:08:58.238326 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:58.238331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:58.238430 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:58.238541 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:58.239157 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.240231 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.241993 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.242619 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.242725 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:58.244131 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:58.244164 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:58.244180 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:58.244971 (+ 791us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:58.245028 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:58.245052 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:58.245127 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:58.245370 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 15:08:58.245478 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:08:58.246958 8887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:58.248781 9196 ts_tablet_manager.cc:1918] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:08:58.249002 9196 log.cc:1196] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/9af227cf49c3401c8f2289456bd8dccd
I20260925 15:08:58.249603 9196 ts_tablet_manager.cc:1939] T 9af227cf49c3401c8f2289456bd8dccd P 17682c7576b54f92bbc1c998fe9c0a29: Deleting consensus metadata
I20260925 15:08:58.250427 9197 ts_tablet_manager.cc:1918] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:08:58.250577 9197 log.cc:1196] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/9af227cf49c3401c8f2289456bd8dccd
I20260925 15:08:58.251037 9197 ts_tablet_manager.cc:1939] T 9af227cf49c3401c8f2289456bd8dccd P 61e238aa7fa84f0d85a3629d03473b81: Deleting consensus metadata
I20260925 15:08:58.251390 8693 catalog_manager.cc:4841] TS 17682c7576b54f92bbc1c998fe9c0a29 (127.22.236.2:44357): tablet 9af227cf49c3401c8f2289456bd8dccd (table test-table [id=76c9e9b61e4242cc843d91872787ace6]) successfully deleted
I20260925 15:08:58.252624 8693 catalog_manager.cc:4841] TS 61e238aa7fa84f0d85a3629d03473b81 (127.22.236.3:35773): tablet 9af227cf49c3401c8f2289456bd8dccd (table test-table [id=76c9e9b61e4242cc843d91872787ace6]) successfully deleted
I20260925 15:08:58.340111 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 8765
W20260925 15:08:58.354604 8691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.236.1:38525 (error 108)
I20260925 15:08:58.354786 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 8896
I20260925 15:08:58.356498 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.355939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38525 (local address 127.0.0.1:38646)
0925 15:08:58.356124 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:58.356264 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:08:58.356846 8691 catalog_manager.cc:4584] TS 6768e5a1508a41399bc97a905cb7599c (127.22.236.1:38525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9af227cf49c3401c8f2289456bd8dccd: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38525: connect: Connection refused (error 111)
I20260925 15:08:58.361378 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.355623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34576 (local address 127.22.236.2:44357)
0925 15:08:58.355792 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:58.355798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:58.355818 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:08:58.355951 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:58.355957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:58.356065 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:58.356193 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:58.356660 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.357528 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.358790 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.359174 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.359263 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:58.360216 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:58.360224 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:58.360237 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:58.360775 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:58.360820 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:58.360839 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:58.360921 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:58.361018 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 15:08:58.361101 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:08:58.361997 9018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:58.410413 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.409857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38525 (local address 127.0.0.1:38654)
0925 15:08:58.410103 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:58.410214 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 15:08:58.411443 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.411049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38525 (local address 127.0.0.1:38660)
0925 15:08:58.411186 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:58.411278 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:08:58.450654 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 8896
I20260925 15:08:58.464273 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 9026
I20260925 15:08:58.470647 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.465062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.22.236.3:35773)
0925 15:08:58.465239 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:58.465244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:58.465262 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:08:58.465324 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:58.465328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:58.465420 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:58.465502 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:58.465818 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.466569 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.467728 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:58.468116 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:58.468252 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:58.469269 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:58.469279 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:58.469296 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:58.469888 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:58.469943 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:58.469965 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:58.470062 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:58.470209 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:08:58.470320 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:08:58.471330 9148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:58.471882 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.471357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38525 (local address 127.0.0.1:38664)
0925 15:08:58.471593 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:58.471707 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 15:08:58.472951 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.472581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38525 (local address 127.0.0.1:38674)
0925 15:08:58.472711 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:58.472795 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 15:08:58.553720 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.553161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38525 (local address 127.0.0.1:38686)
0925 15:08:58.553402 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:58.553536 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 15:08:58.554970 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:58.554483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38525 (local address 127.0.0.1:38700)
0925 15:08:58.554718 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:58.554807 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260925 15:08:58.559229 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 9026
I20260925 15:08:58.571656 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 8675
I20260925 15:08:58.572682 8736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:58.679010 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 8675
2026-09-25T15:08:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2602 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:08:58 dist-test-slave-nsrc krb5kdc[9206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:08:58 dist-test-slave-nsrc krb5kdc[9206](info): set up 2 sockets
Sep 25 15:08:58 dist-test-slave-nsrc krb5kdc[9206](info): commencing operation
krb5kdc: starting...
W20260925 15:08:59.688174 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:08:59 dist-test-slave-nsrc krb5kdc[9206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:08:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:08:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:08:59.906973 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:40263
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:40263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:00.221060 9219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:00.221331 9219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:00.221462 9219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:00.228606 9219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:00.228713 9219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:00.228789 9219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:00.228866 9219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:00.228935 9219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:00.238685 9219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:40263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:40263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:00.239981 9219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:00.241291 9219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:00.252705 9225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:00.252657 9227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:00.253173 9224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:00.253630 9219 server_base.cc:1034] running on GCE node
I20260925 15:09:00.254348 9219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:00.255733 9219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:00.256942 9219 hybrid_clock.cc:648] HybridClock initialized: now 1790348940256884 us; error 36 us; skew 500 ppm
I20260925 15:09:00.257277 9219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:00 dist-test-slave-nsrc krb5kdc[9206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348940, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:00.260257 9219 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:09:00.261718 9219 webserver.cc:460] Webserver started at http://127.22.236.62:37561/ using document root <none> and password file <none>
I20260925 15:09:00.262398 9219 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:00.262517 9219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:00.262795 9219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:00.265798 9219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "53c06fb9e8024dd9a576aed027f09d0e"
format_stamp: "Formatted at 2026-09-25 15:09:00 on dist-test-slave-nsrc"
server_key: "130e13df9d0928d9979d56b8be5b855d"
server_key_iv: "b6bd9343e7f7216dec29db1d966f882e"
server_key_version: "encryptionkey@0"
I20260925 15:09:00.266530 9219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "53c06fb9e8024dd9a576aed027f09d0e"
format_stamp: "Formatted at 2026-09-25 15:09:00 on dist-test-slave-nsrc"
server_key: "130e13df9d0928d9979d56b8be5b855d"
server_key_iv: "b6bd9343e7f7216dec29db1d966f882e"
server_key_version: "encryptionkey@0"
I20260925 15:09:00.271716 9219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 15:09:00.275280 9234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:00.276688 9219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:00.276852 9219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "53c06fb9e8024dd9a576aed027f09d0e"
format_stamp: "Formatted at 2026-09-25 15:09:00 on dist-test-slave-nsrc"
server_key: "130e13df9d0928d9979d56b8be5b855d"
server_key_iv: "b6bd9343e7f7216dec29db1d966f882e"
server_key_version: "encryptionkey@0"
I20260925 15:09:00.277041 9219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:00.322449 9219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:00.326449 9219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:00.326720 9219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:00.343689 9219 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:40263
I20260925 15:09:00.343703 9285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:40263 every 8 connection(s)
I20260925 15:09:00.345049 9219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:00.347429 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 9219
I20260925 15:09:00.347582 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:00.348021 23472 external_mini_cluster.cc:1428] Setting key 392439f5b72302f3bdb77c929471af77
I20260925 15:09:00.348865 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:09:00 dist-test-slave-nsrc krb5kdc[9206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:09:00.363054 9286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e: Bootstrap starting.
I20260925 15:09:00.366308 9286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:00.367517 9286 log.cc:824] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:00.370518 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:00.349827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.22.236.62:40263)
0925 15:09:00.350543 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:00.350558 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:00.350599 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:09:00.352086 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:00.352108 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:00.352537 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:00.352864 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:00.354653 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:00.356000 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:00.357845 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:00.358537 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:00.358676 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:00.363535 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:00.363543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:00.363559 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:00.363604 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:00.366499 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:00.367291 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:00.367297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:00.367304 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:00.367477 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:00.367854 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:00.367857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:00.367859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:00.368596 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:00.368891 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:00.369202 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 15:09:00.369537 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":302,"threads_started":1}
I20260925 15:09:00.370605 9286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e: No bootstrap required, opened a new log
I20260925 15:09:00.377939 9286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c06fb9e8024dd9a576aed027f09d0e" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 40263 } }
I20260925 15:09:00.378297 9286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:00.378391 9286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c06fb9e8024dd9a576aed027f09d0e, State: Initialized, Role: FOLLOWER
I20260925 15:09:00.379019 9286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c06fb9e8024dd9a576aed027f09d0e" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 40263 } }
I20260925 15:09:00.379254 9286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:00.379388 9286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:00.379544 9286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:00.381713 9286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c06fb9e8024dd9a576aed027f09d0e" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 40263 } }
I20260925 15:09:00.382210 9286 leader_election.cc:302] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53c06fb9e8024dd9a576aed027f09d0e; no voters:
I20260925 15:09:00.382720 9286 leader_election.cc:288] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:00.382840 9291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:00.383450 9291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [term 1 LEADER]: Becoming Leader. State: Replica: 53c06fb9e8024dd9a576aed027f09d0e, State: Running, Role: LEADER
I20260925 15:09:00.383967 9291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c06fb9e8024dd9a576aed027f09d0e" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 40263 } }
I20260925 15:09:00.384654 9286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:00.386880 9293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53c06fb9e8024dd9a576aed027f09d0e. Latest consensus state: current_term: 1 leader_uuid: "53c06fb9e8024dd9a576aed027f09d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c06fb9e8024dd9a576aed027f09d0e" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 40263 } } }
I20260925 15:09:00.386847 9292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53c06fb9e8024dd9a576aed027f09d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c06fb9e8024dd9a576aed027f09d0e" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 40263 } } }
I20260925 15:09:00.387367 9292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:00.387368 9293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:00.392740 9296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:00.397966 9296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:00.416042 9296 catalog_manager.cc:1347] Generated new cluster ID: 1f75b2a44c634faa923d7810fb6f034a
I20260925 15:09:00.416201 9296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:00.438710 9296 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:00.440027 9296 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:00.447633 9296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e: Generated new TSK 0
I20260925 15:09:00.448460 9296 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:09:00.507710 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:40263
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:00.825166 9312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:00.825471 9312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:00.825613 9312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:00.832828 9312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:00.832953 9312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:00.833102 9312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:09:00.842768 9312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:40263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:00.844012 9312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:00.845357 9312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:00.857887 9318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:00.857940 9317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:00.857924 9320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:00.858659 9312 server_base.cc:1034] running on GCE node
I20260925 15:09:00.859277 9312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:00.860194 9312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:00.861410 9312 hybrid_clock.cc:648] HybridClock initialized: now 1790348940861360 us; error 41 us; skew 500 ppm
I20260925 15:09:00.861719 9312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:00 dist-test-slave-nsrc krb5kdc[9206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348940, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:00.864655 9312 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:09:00.866096 9312 webserver.cc:460] Webserver started at http://127.22.236.1:33931/ using document root <none> and password file <none>
I20260925 15:09:00.866719 9312 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:00.866842 9312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:00.867132 9312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:00.870015 9312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "143657861f814ae38233cf9a46f20421"
format_stamp: "Formatted at 2026-09-25 15:09:00 on dist-test-slave-nsrc"
server_key: "1a62aa6e55b864c153bbfca4d8f2f508"
server_key_iv: "22d688a497d5b7e586cd7cd380396fb4"
server_key_version: "encryptionkey@0"
I20260925 15:09:00.870711 9312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "143657861f814ae38233cf9a46f20421"
format_stamp: "Formatted at 2026-09-25 15:09:00 on dist-test-slave-nsrc"
server_key: "1a62aa6e55b864c153bbfca4d8f2f508"
server_key_iv: "22d688a497d5b7e586cd7cd380396fb4"
server_key_version: "encryptionkey@0"
I20260925 15:09:00.875569 9312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:09:00.878847 9327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:00.880100 9312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:09:00.880275 9312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "143657861f814ae38233cf9a46f20421"
format_stamp: "Formatted at 2026-09-25 15:09:00 on dist-test-slave-nsrc"
server_key: "1a62aa6e55b864c153bbfca4d8f2f508"
server_key_iv: "22d688a497d5b7e586cd7cd380396fb4"
server_key_version: "encryptionkey@0"
I20260925 15:09:00.880450 9312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:00.899116 9312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:00.903123 9312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:00.903398 9312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:00.904347 9312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:00.905648 9312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:00.905725 9312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:00.905850 9312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:00.905903 9312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:00.934984 9312 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:35435
I20260925 15:09:00.935034 9439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:35435 every 8 connection(s)
I20260925 15:09:00.936232 9312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:09:00.942238 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 9312
I20260925 15:09:00.942454 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:09:00.943005 23472 external_mini_cluster.cc:1428] Setting key 304880447f924eeb7991d68ef2d8df22
Sep 25 15:09:00 dist-test-slave-nsrc krb5kdc[9206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348940, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:09:00.957104 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:00.938676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:58873 (local address 127.22.236.62:40263)
0925 15:09:00.939130 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:00.939135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:00.940088 (+ 953us) server_negotiation.cc:407] Connection header received
0925 15:09:00.941848 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:00.941852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:00.941918 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:00.942008 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:00.943939 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:00.944777 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:00.946941 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:00.947357 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:00.947463 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:00.951373 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:00.951378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:00.951381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:00.951425 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:00.954110 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:00.954779 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:00.954785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:00.954788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:00.954927 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:00.955380 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:00.955385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:00.955388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:00.955731 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:00.955863 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:00.956549 (+ 686us) server_negotiation.cc:299] Negotiation successful
0925 15:09:00.956753 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":213,"threads_started":1}
I20260925 15:09:00.957971 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:00.939206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:40263 (local address 127.22.236.1:58873)
0925 15:09:00.939929 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:00.939976 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:00.941600 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:00.942273 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:00.942285 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:00.943021 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:00.943744 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:00.943771 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:00.944931 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:00.944936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:00.946802 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:00.946817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:00.947560 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:00.947566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:00.947828 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:00.948567 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:00.948599 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:00.951126 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:00.954238 (+ 3112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:00.954245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:00.954259 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:00.954667 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:00.955078 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:00.955082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:00.955085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:00.955283 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:00.955834 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:00.955840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:00.956206 (+ 366us) client_negotiation.cc:769] Sending connection context
0925 15:09:00.956453 (+ 247us) client_negotiation.cc:240] Negotiation successful
0925 15:09:00.956849 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":265,"threads_started":1}
I20260925 15:09:00.959352 9440 heartbeater.cc:347] Connected to a master server at 127.22.236.62:40263
I20260925 15:09:00.959671 9440 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:00.960518 9440 heartbeater.cc:511] Master 127.22.236.62:40263 requested a full tablet report, sending...
I20260925 15:09:00.962558 9251 ts_manager.cc:194] Registered new tserver with Master: 143657861f814ae38233cf9a46f20421 (127.22.236.1:35435)
I20260925 15:09:00.964092 9251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:58873
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:09:00.995079 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:40263
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:01.309335 9446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:01.309613 9446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:01.309746 9446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:01.317039 9446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:01.317173 9446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:01.317334 9446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:09:01.327026 9446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:40263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:01.328298 9446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:01.329572 9446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:01.342063 9451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:01.342142 9452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:01.342161 9454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:01.343120 9446 server_base.cc:1034] running on GCE node
I20260925 15:09:01.343900 9446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:01.345077 9446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:01.346287 9446 hybrid_clock.cc:648] HybridClock initialized: now 1790348941346221 us; error 62 us; skew 500 ppm
I20260925 15:09:01.346593 9446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:01 dist-test-slave-nsrc krb5kdc[9206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348941, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:01.349591 9446 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:09:01.353274 9446 webserver.cc:460] Webserver started at http://127.22.236.2:45671/ using document root <none> and password file <none>
I20260925 15:09:01.353967 9446 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:01.354074 9446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:01.354369 9446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:01.357239 9446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "4560488035d74e20aa05df007b4f1eaf"
format_stamp: "Formatted at 2026-09-25 15:09:01 on dist-test-slave-nsrc"
server_key: "14bc44095c61ea97a57cb145be436d46"
server_key_iv: "4e20f3482ec94431c7be2c496a9b9aef"
server_key_version: "encryptionkey@0"
I20260925 15:09:01.357976 9446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "4560488035d74e20aa05df007b4f1eaf"
format_stamp: "Formatted at 2026-09-25 15:09:01 on dist-test-slave-nsrc"
server_key: "14bc44095c61ea97a57cb145be436d46"
server_key_iv: "4e20f3482ec94431c7be2c496a9b9aef"
server_key_version: "encryptionkey@0"
I20260925 15:09:01.362941 9446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 15:09:01.366298 9461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:01.367540 9446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:09:01.367673 9446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "4560488035d74e20aa05df007b4f1eaf"
format_stamp: "Formatted at 2026-09-25 15:09:01 on dist-test-slave-nsrc"
server_key: "14bc44095c61ea97a57cb145be436d46"
server_key_iv: "4e20f3482ec94431c7be2c496a9b9aef"
server_key_version: "encryptionkey@0"
I20260925 15:09:01.367843 9446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:01.390136 9446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:01.394404 9446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:01.394675 9446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:01.395643 9446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:01.397013 9446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:01.397089 9446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:01.397207 9446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:01.397264 9446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:01.426487 9446 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:33699
I20260925 15:09:01.426530 9573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:33699 every 8 connection(s)
I20260925 15:09:01.427752 9446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:09:01.437757 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 9446
I20260925 15:09:01.437940 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:09:01.438390 23472 external_mini_cluster.cc:1428] Setting key 3e966e23764bc0bd8f569b6f9469476c
Sep 25 15:09:01 dist-test-slave-nsrc krb5kdc[9206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348941, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:09:01.446744 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:01.430148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:39495 (local address 127.22.236.62:40263)
0925 15:09:01.430311 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:01.430315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:01.431362 (+ 1047us) server_negotiation.cc:407] Connection header received
0925 15:09:01.433153 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:01.433158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:01.433240 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:01.433329 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:01.434728 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:01.435472 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:01.437071 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:01.437558 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:01.437707 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:01.441266 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:01.441271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:01.441275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:01.441307 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:01.443797 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:01.444550 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:01.444555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:01.444557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:01.444687 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:01.445094 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:01.445097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:01.445099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:01.445355 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:01.445482 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:01.446185 (+ 703us) server_negotiation.cc:299] Negotiation successful
0925 15:09:01.446406 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:09:01.447700 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:01.430536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:40263 (local address 127.22.236.2:39495)
0925 15:09:01.431193 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:01.431236 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:01.432889 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:01.433514 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:01.433522 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:01.434012 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:01.434567 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:01.434585 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:01.435637 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:01.435641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:01.436956 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:01.436970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:01.437808 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:01.437817 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:01.438059 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:01.438701 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:01.438725 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:01.441066 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:01.443945 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:01.443954 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:01.443972 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:01.444433 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:01.444805 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:01.444810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:01.444812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:01.444983 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:01.445472 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:01.445480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:01.445827 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 15:09:01.446061 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 15:09:01.446603 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":247,"threads_started":1}
I20260925 15:09:01.449100 9574 heartbeater.cc:347] Connected to a master server at 127.22.236.62:40263
I20260925 15:09:01.449379 9574 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:01.450037 9574 heartbeater.cc:511] Master 127.22.236.62:40263 requested a full tablet report, sending...
I20260925 15:09:01.451403 9251 ts_manager.cc:194] Registered new tserver with Master: 4560488035d74e20aa05df007b4f1eaf (127.22.236.2:33699)
I20260925 15:09:01.452428 9251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:39495
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:09:01.486425 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:40263
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:01.796090 9579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:01.796397 9579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:01.796494 9579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:01.803428 9579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:01.803527 9579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:01.803639 9579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:09:01.813102 9579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:40263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:01.814257 9579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:01.815464 9579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:01.827756 9584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:01.827692 9585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:01.827796 9587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:01.828689 9579 server_base.cc:1034] running on GCE node
I20260925 15:09:01.829285 9579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:01.830156 9579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:01.831363 9579 hybrid_clock.cc:648] HybridClock initialized: now 1790348941831308 us; error 47 us; skew 500 ppm
I20260925 15:09:01.831660 9579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:01 dist-test-slave-nsrc krb5kdc[9206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348941, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:01.838141 9579 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:09:01.839609 9579 webserver.cc:460] Webserver started at http://127.22.236.3:38973/ using document root <none> and password file <none>
I20260925 15:09:01.840329 9579 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:01.840449 9579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:01.840736 9579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:01.843562 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "5fd4c761d6444a3894202d0ed3b12ff2"
format_stamp: "Formatted at 2026-09-25 15:09:01 on dist-test-slave-nsrc"
server_key: "e1f97ce84c26537cf69661f880d8bbf0"
server_key_iv: "f1df47f3d2a0e53dd8b5733c0a2e5dd4"
server_key_version: "encryptionkey@0"
I20260925 15:09:01.844478 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "5fd4c761d6444a3894202d0ed3b12ff2"
format_stamp: "Formatted at 2026-09-25 15:09:01 on dist-test-slave-nsrc"
server_key: "e1f97ce84c26537cf69661f880d8bbf0"
server_key_iv: "f1df47f3d2a0e53dd8b5733c0a2e5dd4"
server_key_version: "encryptionkey@0"
I20260925 15:09:01.849433 9579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:09:01.852941 9594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:01.854182 9579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:09:01.854341 9579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "5fd4c761d6444a3894202d0ed3b12ff2"
format_stamp: "Formatted at 2026-09-25 15:09:01 on dist-test-slave-nsrc"
server_key: "e1f97ce84c26537cf69661f880d8bbf0"
server_key_iv: "f1df47f3d2a0e53dd8b5733c0a2e5dd4"
server_key_version: "encryptionkey@0"
I20260925 15:09:01.854516 9579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:01.888082 9579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:01.892338 9579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:01.892601 9579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:01.893646 9579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:01.895063 9579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:01.895140 9579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:01.895267 9579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:01.895330 9579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:01.924578 9579 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:38905
I20260925 15:09:01.924634 9706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:38905 every 8 connection(s)
I20260925 15:09:01.925841 9579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:09:01.927538 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 9579
I20260925 15:09:01.927712 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:09:01.928323 23472 external_mini_cluster.cc:1428] Setting key cbd356c2660c7956dcbc4bd2aaf291da
Sep 25 15:09:01 dist-test-slave-nsrc krb5kdc[9206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348941, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:09:01.945430 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:01.928279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:36103 (local address 127.22.236.62:40263)
0925 15:09:01.928458 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:01.928465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:01.929915 (+ 1450us) server_negotiation.cc:407] Connection header received
0925 15:09:01.931787 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:01.931791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:01.931878 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:01.931964 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:01.933369 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:01.934093 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:01.935568 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:01.936271 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:01.936396 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:01.940070 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:01.940076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:01.940079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:01.940113 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:01.942618 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:01.943318 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:01.943322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:01.943324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:01.943438 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:01.943910 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:01.943913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:01.943914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:01.944231 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:01.944347 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:01.945012 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 15:09:01.945189 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:09:01.946272 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:01.928930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:40263 (local address 127.22.236.3:36103)
0925 15:09:01.929735 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:01.929794 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:01.931550 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:01.932195 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:01.932208 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:01.932686 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:01.933244 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:01.933261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:01.934210 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:01.934214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:01.935432 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:01.935444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:01.936453 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:01.936458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:01.936644 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:01.937252 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:01.937290 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:01.939804 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:01.942742 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:01.942751 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:01.942766 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:01.943221 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:01.943516 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:01.943519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:01.943521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:01.943767 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:01.944355 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:01.944361 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:01.944697 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 15:09:01.944932 (+ 235us) client_negotiation.cc:240] Negotiation successful
0925 15:09:01.945287 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":338,"threads_started":1}
I20260925 15:09:01.947420 9707 heartbeater.cc:347] Connected to a master server at 127.22.236.62:40263
I20260925 15:09:01.947686 9707 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:01.948408 9707 heartbeater.cc:511] Master 127.22.236.62:40263 requested a full tablet report, sending...
I20260925 15:09:01.949739 9251 ts_manager.cc:194] Registered new tserver with Master: 5fd4c761d6444a3894202d0ed3b12ff2 (127.22.236.3:38905)
I20260925 15:09:01.950537 9251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:36103
I20260925 15:09:01.954955 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:09:01.967605 9440 heartbeater.cc:503] Master 127.22.236.62:40263 was elected leader, sending a full tablet report...
I20260925 15:09:01.971205 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:01.958834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.22.236.62:40263)
0925 15:09:01.959031 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:01.959037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:01.959355 (+ 318us) server_negotiation.cc:407] Connection header received
0925 15:09:01.959521 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:01.959525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:01.959584 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:01.959661 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:01.961054 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:01.961746 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:01.963032 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:01.963446 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:01.963553 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:01.966151 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:01.966156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:01.966160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:01.966192 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:01.968844 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:01.969625 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:01.969629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:01.969631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:01.969755 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:01.970098 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:01.970101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:01.970103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:01.970382 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:01.970571 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:01.970842 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 15:09:01.970981 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:09:01.974773 9251 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:09:01.981528 9251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:09:02.017298 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:01.994148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:33699 (local address 127.0.0.1:60162)
0925 15:09:01.994685 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:01.994701 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:01.994802 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:02.009844 (+ 15042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:02.009848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:02.009873 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:02.010359 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:02.010372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.011819 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.011825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:02.014261 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:02.014276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.016220 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.016227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:02.016419 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:02.016493 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 15:09:02.016602 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 15:09:02.016799 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":43,"thread_start_us":291,"threads_started":1}
I20260925 15:09:02.017514 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:01.994587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.22.236.2:33699)
0925 15:09:02.009040 (+ 14453us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:02.009046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:02.009070 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:09:02.009150 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:02.009156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:02.009409 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:02.009847 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:02.010526 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.011699 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.014456 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.015969 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.016811 (+ 842us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:02.016992 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:02.017072 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 15:09:02.017244 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14330,"thread_start_us":14258,"threads_started":1}
I20260925 15:09:02.017398 9719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:01.992967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:38905 (local address 127.0.0.1:53644)
0925 15:09:01.993799 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:01.993848 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:01.994036 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:02.010569 (+ 16533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:02.010573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:02.010600 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:02.010910 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:02.010921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.012643 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.012646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:02.014639 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:02.014652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.016541 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.016547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:02.016710 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:02.016742 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:09:02.016831 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:09:02.016971 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":8,"thread_start_us":398,"threads_started":1}
I20260925 15:09:02.017887 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:01.995330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55830 (local address 127.22.236.1:35435)
0925 15:09:02.008783 (+ 13453us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:02.008794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:02.008830 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:09:02.008937 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:02.008946 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:02.009248 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:02.009484 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:02.010658 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.012115 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.014488 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.016351 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.016489 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:02.016701 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:02.017400 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 15:09:02.017580 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13244,"thread_start_us":13049,"threads_started":1}
I20260925 15:09:02.017985 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:01.993179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.22.236.3:38905)
0925 15:09:02.009523 (+ 16344us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:02.009531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:02.009563 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:09:02.009656 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:02.009663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:02.010395 (+ 732us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:02.010589 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:02.011085 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.012507 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.014787 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.016301 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.016462 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:02.016698 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:02.017545 (+ 847us) server_negotiation.cc:299] Negotiation successful
0925 15:09:02.017703 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16166,"thread_start_us":16383,"threads_started":1}
I20260925 15:09:02.018437 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:01.994753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:35435 (local address 127.0.0.1:55830)
0925 15:09:02.005488 (+ 10735us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:02.005511 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:02.005645 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:02.009797 (+ 4152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:02.009805 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:02.009833 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:02.010451 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:02.010464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.012265 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.012270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:02.014326 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:02.014336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.016838 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.016844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:02.017009 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:02.017041 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:09:02.017133 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:09:02.017259 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10564,"mutex_wait_us":58,"thread_start_us":10393,"threads_started":1}
I20260925 15:09:02.021556 9375 tablet_service.cc:1467] Processing CreateTablet for tablet 323d35351b244ca29b83ac913f2f6159 (DEFAULT_TABLE table=test-table [id=3a363e477140467db519db3ecb7dbb9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:02.021729 9642 tablet_service.cc:1467] Processing CreateTablet for tablet 323d35351b244ca29b83ac913f2f6159 (DEFAULT_TABLE table=test-table [id=3a363e477140467db519db3ecb7dbb9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:02.021713 9509 tablet_service.cc:1467] Processing CreateTablet for tablet 323d35351b244ca29b83ac913f2f6159 (DEFAULT_TABLE table=test-table [id=3a363e477140467db519db3ecb7dbb9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:02.023093 9642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323d35351b244ca29b83ac913f2f6159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:02.023093 9375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323d35351b244ca29b83ac913f2f6159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:02.023183 9509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323d35351b244ca29b83ac913f2f6159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:02.034574 9725 tablet_bootstrap.cc:492] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421: Bootstrap starting.
I20260925 15:09:02.036863 9726 tablet_bootstrap.cc:492] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf: Bootstrap starting.
I20260925 15:09:02.037680 9727 tablet_bootstrap.cc:492] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2: Bootstrap starting.
I20260925 15:09:02.038371 9725 tablet_bootstrap.cc:654] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:02.039665 9725 log.cc:824] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:02.040064 9726 tablet_bootstrap.cc:654] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:02.040475 9727 tablet_bootstrap.cc:654] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:02.041129 9726 log.cc:824] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:02.041507 9727 log.cc:824] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:02.043534 9725 tablet_bootstrap.cc:492] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421: No bootstrap required, opened a new log
I20260925 15:09:02.043810 9725 ts_tablet_manager.cc:1397] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260925 15:09:02.044468 9726 tablet_bootstrap.cc:492] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf: No bootstrap required, opened a new log
I20260925 15:09:02.044730 9726 ts_tablet_manager.cc:1397] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:09:02.050504 9727 tablet_bootstrap.cc:492] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2: No bootstrap required, opened a new log
I20260925 15:09:02.050801 9727 ts_tablet_manager.cc:1397] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20260925 15:09:02.054111 9725 raft_consensus.cc:348] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd4c761d6444a3894202d0ed3b12ff2" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38905 } } peers { permanent_uuid: "4560488035d74e20aa05df007b4f1eaf" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33699 } } peers { permanent_uuid: "143657861f814ae38233cf9a46f20421" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35435 } }
I20260925 15:09:02.054603 9725 raft_consensus.cc:374] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:02.054718 9725 raft_consensus.cc:729] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143657861f814ae38233cf9a46f20421, State: Initialized, Role: FOLLOWER
I20260925 15:09:02.054592 9726 raft_consensus.cc:348] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd4c761d6444a3894202d0ed3b12ff2" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38905 } } peers { permanent_uuid: "4560488035d74e20aa05df007b4f1eaf" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33699 } } peers { permanent_uuid: "143657861f814ae38233cf9a46f20421" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35435 } }
I20260925 15:09:02.054997 9726 raft_consensus.cc:374] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:02.055356 9725 consensus_queue.cc:260] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd4c761d6444a3894202d0ed3b12ff2" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38905 } } peers { permanent_uuid: "4560488035d74e20aa05df007b4f1eaf" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33699 } } peers { permanent_uuid: "143657861f814ae38233cf9a46f20421" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35435 } }
I20260925 15:09:02.056676 9726 raft_consensus.cc:729] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4560488035d74e20aa05df007b4f1eaf, State: Initialized, Role: FOLLOWER
I20260925 15:09:02.057219 9725 ts_tablet_manager.cc:1428] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 15:09:02.057394 9726 consensus_queue.cc:260] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd4c761d6444a3894202d0ed3b12ff2" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38905 } } peers { permanent_uuid: "4560488035d74e20aa05df007b4f1eaf" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33699 } } peers { permanent_uuid: "143657861f814ae38233cf9a46f20421" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35435 } }
I20260925 15:09:02.058508 9574 heartbeater.cc:503] Master 127.22.236.62:40263 was elected leader, sending a full tablet report...
I20260925 15:09:02.058866 9727 raft_consensus.cc:348] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd4c761d6444a3894202d0ed3b12ff2" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38905 } } peers { permanent_uuid: "4560488035d74e20aa05df007b4f1eaf" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33699 } } peers { permanent_uuid: "143657861f814ae38233cf9a46f20421" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35435 } }
I20260925 15:09:02.059150 9726 ts_tablet_manager.cc:1428] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260925 15:09:02.059262 9727 raft_consensus.cc:374] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:02.059355 9727 raft_consensus.cc:729] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd4c761d6444a3894202d0ed3b12ff2, State: Initialized, Role: FOLLOWER
I20260925 15:09:02.059896 9727 consensus_queue.cc:260] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd4c761d6444a3894202d0ed3b12ff2" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38905 } } peers { permanent_uuid: "4560488035d74e20aa05df007b4f1eaf" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33699 } } peers { permanent_uuid: "143657861f814ae38233cf9a46f20421" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35435 } }
I20260925 15:09:02.061012 9707 heartbeater.cc:503] Master 127.22.236.62:40263 was elected leader, sending a full tablet report...
I20260925 15:09:02.061585 9727 ts_tablet_manager.cc:1428] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260925 15:09:02.165983 9732 raft_consensus.cc:482] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:09:02.166280 9732 raft_consensus.cc:504] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd4c761d6444a3894202d0ed3b12ff2" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38905 } } peers { permanent_uuid: "4560488035d74e20aa05df007b4f1eaf" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33699 } } peers { permanent_uuid: "143657861f814ae38233cf9a46f20421" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35435 } }
I20260925 15:09:02.167906 9732 leader_election.cc:288] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fd4c761d6444a3894202d0ed3b12ff2 (127.22.236.3:38905), 143657861f814ae38233cf9a46f20421 (127.22.236.1:35435)
I20260925 15:09:02.175285 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:02.168386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:48283 (local address 127.22.236.3:38905)
0925 15:09:02.168603 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:02.168610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:02.168823 (+ 213us) server_negotiation.cc:407] Connection header received
0925 15:09:02.169033 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:02.169040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:02.169118 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:02.169202 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:02.169890 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.170769 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.173202 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.174370 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.174538 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:02.174613 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:02.174910 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 15:09:02.175032 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:09:02.175311 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:02.169098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:40401 (local address 127.22.236.1:35435)
0925 15:09:02.169301 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:02.169306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:02.169496 (+ 190us) server_negotiation.cc:407] Connection header received
0925 15:09:02.169642 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:02.169648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:02.169722 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:02.169824 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:02.170353 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.171094 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.173340 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.174501 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.174630 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:02.174687 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:02.174988 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 15:09:02.175101 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:09:02.175405 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:02.168792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:35435 (local address 127.22.236.2:40401)
0925 15:09:02.169394 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:02.169415 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:02.169527 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:02.169833 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:02.169838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:02.169859 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:02.170198 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:02.170209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.171218 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.171223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:02.173201 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:02.173212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.174671 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.174677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:02.174846 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:02.174875 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:09:02.174954 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:09:02.175049 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":312,"threads_started":1}
I20260925 15:09:02.175464 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:02.168165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:38905 (local address 127.22.236.2:48283)
0925 15:09:02.168727 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:02.168741 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:02.168893 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:02.169324 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:02.169329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:02.169359 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:02.169754 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:02.169766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.170898 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.170904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:02.173049 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:02.173064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.174562 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.174568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:02.174747 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:02.174783 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:09:02.174877 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:09:02.175009 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"spinlock_wait_cycles":21888,"thread_start_us":278,"threads_started":1}
I20260925 15:09:02.176445 9662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323d35351b244ca29b83ac913f2f6159" candidate_uuid: "4560488035d74e20aa05df007b4f1eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd4c761d6444a3894202d0ed3b12ff2" is_pre_election: true
I20260925 15:09:02.176445 9395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323d35351b244ca29b83ac913f2f6159" candidate_uuid: "4560488035d74e20aa05df007b4f1eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143657861f814ae38233cf9a46f20421" is_pre_election: true
I20260925 15:09:02.177003 9395 raft_consensus.cc:2454] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4560488035d74e20aa05df007b4f1eaf in term 0.
I20260925 15:09:02.177003 9662 raft_consensus.cc:2454] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4560488035d74e20aa05df007b4f1eaf in term 0.
W20260925 15:09:02.177340 9708 tablet.cc:2369] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:09:02.177719 9465 leader_election.cc:302] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143657861f814ae38233cf9a46f20421, 4560488035d74e20aa05df007b4f1eaf; no voters:
I20260925 15:09:02.178083 9732 raft_consensus.cc:2781] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:09:02.178213 9732 raft_consensus.cc:482] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:09:02.178306 9732 raft_consensus.cc:3037] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [term 0 FOLLOWER]: Advancing to term 1
W20260925 15:09:02.179499 9575 tablet.cc:2369] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:09:02.180572 9732 raft_consensus.cc:504] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd4c761d6444a3894202d0ed3b12ff2" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38905 } } peers { permanent_uuid: "4560488035d74e20aa05df007b4f1eaf" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33699 } } peers { permanent_uuid: "143657861f814ae38233cf9a46f20421" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35435 } }
I20260925 15:09:02.181115 9732 leader_election.cc:288] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [CANDIDATE]: Term 1 election: Requested vote from peers 5fd4c761d6444a3894202d0ed3b12ff2 (127.22.236.3:38905), 143657861f814ae38233cf9a46f20421 (127.22.236.1:35435)
I20260925 15:09:02.181591 9662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323d35351b244ca29b83ac913f2f6159" candidate_uuid: "4560488035d74e20aa05df007b4f1eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd4c761d6444a3894202d0ed3b12ff2"
I20260925 15:09:02.181646 9395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323d35351b244ca29b83ac913f2f6159" candidate_uuid: "4560488035d74e20aa05df007b4f1eaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143657861f814ae38233cf9a46f20421"
I20260925 15:09:02.181829 9662 raft_consensus.cc:3037] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:02.181828 9395 raft_consensus.cc:3037] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:02.184993 9395 raft_consensus.cc:2454] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4560488035d74e20aa05df007b4f1eaf in term 1.
I20260925 15:09:02.184993 9662 raft_consensus.cc:2454] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4560488035d74e20aa05df007b4f1eaf in term 1.
I20260925 15:09:02.185346 9465 leader_election.cc:302] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143657861f814ae38233cf9a46f20421, 4560488035d74e20aa05df007b4f1eaf; no voters:
I20260925 15:09:02.185606 9732 raft_consensus.cc:2781] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:02.186028 9732 raft_consensus.cc:686] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [term 1 LEADER]: Becoming Leader. State: Replica: 4560488035d74e20aa05df007b4f1eaf, State: Running, Role: LEADER
I20260925 15:09:02.186542 9732 consensus_queue.cc:237] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd4c761d6444a3894202d0ed3b12ff2" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38905 } } peers { permanent_uuid: "4560488035d74e20aa05df007b4f1eaf" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33699 } } peers { permanent_uuid: "143657861f814ae38233cf9a46f20421" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35435 } }
W20260925 15:09:02.188454 9441 tablet.cc:2369] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:09:02.191323 9249 catalog_manager.cc:5495] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 4560488035d74e20aa05df007b4f1eaf (127.22.236.2). New cstate: current_term: 1 leader_uuid: "4560488035d74e20aa05df007b4f1eaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd4c761d6444a3894202d0ed3b12ff2" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4560488035d74e20aa05df007b4f1eaf" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 33699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "143657861f814ae38233cf9a46f20421" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35435 } health_report { overall_health: UNKNOWN } } }
I20260925 15:09:02.240213 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:02.235115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.22.236.2:33699)
0925 15:09:02.235307 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:02.235313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:02.235330 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:09:02.235414 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:02.235418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:02.235500 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:02.235625 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:02.236015 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.236746 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.238092 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.238497 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.238600 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:02.239146 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:02.239605 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:02.239757 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:02.239902 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 15:09:02.239955 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:09:02.247704 9395 raft_consensus.cc:1261] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421 [term 1 FOLLOWER]: Refusing update from remote peer 4560488035d74e20aa05df007b4f1eaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:02.247713 9662 raft_consensus.cc:1261] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2 [term 1 FOLLOWER]: Refusing update from remote peer 4560488035d74e20aa05df007b4f1eaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:02.248579 9736 consensus_queue.cc:1035] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "143657861f814ae38233cf9a46f20421" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 35435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:02.248898 9732 consensus_queue.cc:1035] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fd4c761d6444a3894202d0ed3b12ff2" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 38905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:02.272555 9251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:09:02.272804 9251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:09:02.276261 9251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53c06fb9e8024dd9a576aed027f09d0e: Sending DeleteTablet for 3 replicas of tablet 323d35351b244ca29b83ac913f2f6159
I20260925 15:09:02.277383 9642 tablet_service.cc:1514] Processing DeleteTablet for tablet 323d35351b244ca29b83ac913f2f6159 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:02 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:53644
I20260925 15:09:02.277544 9375 tablet_service.cc:1514] Processing DeleteTablet for tablet 323d35351b244ca29b83ac913f2f6159 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:02 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:55830
I20260925 15:09:02.278244 9509 tablet_service.cc:1514] Processing DeleteTablet for tablet 323d35351b244ca29b83ac913f2f6159 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:02 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:60162
I20260925 15:09:02.280761 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 9312
I20260925 15:09:02.289986 9749 tablet_replica.cc:330] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2: stopping tablet replica
I20260925 15:09:02.290226 9750 tablet_replica.cc:330] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421: stopping tablet replica
I20260925 15:09:02.290567 9749 raft_consensus.cc:2229] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:09:02.290709 9750 raft_consensus.cc:2229] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:09:02.291172 9749 raft_consensus.cc:2258] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:02.291231 9750 raft_consensus.cc:2258] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:02.293507 9750 ts_tablet_manager.cc:1905] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:02.293589 9749 ts_tablet_manager.cc:1905] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:02.300643 9750 ts_tablet_manager.cc:1918] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:09:02.300870 9750 log.cc:1196] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/323d35351b244ca29b83ac913f2f6159
I20260925 15:09:02.301175 9749 ts_tablet_manager.cc:1918] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:09:02.301242 9751 tablet_replica.cc:330] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf: stopping tablet replica
I20260925 15:09:02.301465 9749 log.cc:1196] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/323d35351b244ca29b83ac913f2f6159
I20260925 15:09:02.301462 9750 ts_tablet_manager.cc:1939] T 323d35351b244ca29b83ac913f2f6159 P 143657861f814ae38233cf9a46f20421: Deleting consensus metadata
I20260925 15:09:02.301774 9751 raft_consensus.cc:2229] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:09:02.302011 9749 ts_tablet_manager.cc:1939] T 323d35351b244ca29b83ac913f2f6159 P 5fd4c761d6444a3894202d0ed3b12ff2: Deleting consensus metadata
I20260925 15:09:02.302532 9751 raft_consensus.cc:2258] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 15:09:02 dist-test-slave-nsrc krb5kdc[9206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:09:02.303399 9238 catalog_manager.cc:4841] TS 143657861f814ae38233cf9a46f20421 (127.22.236.1:35435): tablet 323d35351b244ca29b83ac913f2f6159 (table test-table [id=3a363e477140467db519db3ecb7dbb9e]) successfully deleted
I20260925 15:09:02.304854 9238 catalog_manager.cc:4841] TS 5fd4c761d6444a3894202d0ed3b12ff2 (127.22.236.3:38905): tablet 323d35351b244ca29b83ac913f2f6159 (table test-table [id=3a363e477140467db519db3ecb7dbb9e]) successfully deleted
I20260925 15:09:02.304953 9751 ts_tablet_manager.cc:1905] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:02.310932 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:02.282256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.22.236.1:35435)
0925 15:09:02.282443 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:02.282449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:02.292419 (+ 9970us) server_negotiation.cc:407] Connection header received
0925 15:09:02.292602 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:02.292608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:02.292696 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:02.292794 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:02.294754 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.295926 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.297768 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.298402 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.298502 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:02.303828 (+ 5326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:02.303842 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:02.303859 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:02.303897 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:02.307423 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:02.308364 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:02.308373 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:02.308376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:02.308535 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:02.309044 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:02.309049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:02.309051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:02.309475 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:02.309735 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:02.310326 (+ 591us) server_negotiation.cc:299] Negotiation successful
0925 15:09:02.310560 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:09:02.312057 9751 ts_tablet_manager.cc:1918] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:09:02.312281 9751 log.cc:1196] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/323d35351b244ca29b83ac913f2f6159
I20260925 15:09:02.312332 9435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:02.312755 9751 ts_tablet_manager.cc:1939] T 323d35351b244ca29b83ac913f2f6159 P 4560488035d74e20aa05df007b4f1eaf: Deleting consensus metadata
I20260925 15:09:02.314072 9238 catalog_manager.cc:4841] TS 4560488035d74e20aa05df007b4f1eaf (127.22.236.2:33699): tablet 323d35351b244ca29b83ac913f2f6159 (table test-table [id=3a363e477140467db519db3ecb7dbb9e]) successfully deleted
I20260925 15:09:02.417047 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 9312
I20260925 15:09:02.434962 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 9446
Sep 25 15:09:02 dist-test-slave-nsrc krb5kdc[9206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:09:02.454212 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:02.435766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.22.236.2:33699)
0925 15:09:02.435961 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:02.435967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:02.436071 (+ 104us) server_negotiation.cc:407] Connection header received
0925 15:09:02.436276 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:02.436281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:02.436369 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:02.436486 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:02.438032 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.439254 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.441318 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.441995 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.442120 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:02.447221 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:02.447233 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:02.447252 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:02.447303 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:02.451206 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:02.452000 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:02.452009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:02.452013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:02.452208 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:02.452577 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:02.452582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:02.452585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:02.453021 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:02.453341 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:02.453613 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 15:09:02.453830 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:09:02.454864 9569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:02.566810 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 9446
I20260925 15:09:02.581540 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 9579
Sep 25 15:09:02 dist-test-slave-nsrc krb5kdc[9206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:09:02.595589 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:02.582197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.22.236.3:38905)
0925 15:09:02.582422 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:02.582428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:02.582449 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:09:02.582572 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:02.582578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:02.582670 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:02.582754 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:02.583871 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.584639 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.585900 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:02.586283 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:02.586369 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:02.590496 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:02.590510 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:02.590525 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:02.590557 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:02.593074 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:02.593697 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:02.593704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:02.593706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:02.593834 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:02.594182 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:02.594185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:02.594187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:02.594528 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:02.594794 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:02.595071 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 15:09:02.595276 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:09:02.596302 9702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:02.682338 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 9579
I20260925 15:09:02.695276 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 9219
I20260925 15:09:02.696311 9281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:02.804576 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 9219
2026-09-25T15:09:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4125 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T15:09:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:02Z Disabled control of system clock
I20260925 15:09:02.871040 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:35087
--webserver_interface=127.22.236.62
--webserver_port=0
--builtin_ntp_servers=127.22.236.20:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:35087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:09:03.185715 9760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:03.186014 9760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:03.186096 9760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:03.193341 9760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:03.193459 9760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:03.193495 9760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:03.193549 9760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:03.193593 9760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:03.202728 9760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:35087
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:35087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:03.203944 9760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:03.205689 9760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:03.216737 9765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:03.216675 9768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:03.216715 9766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:03.217383 9760 server_base.cc:1034] running on GCE node
I20260925 15:09:03.218110 9760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:03.219448 9760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:03.220636 9760 hybrid_clock.cc:648] HybridClock initialized: now 1790348943220587 us; error 47 us; skew 500 ppm
I20260925 15:09:03.220955 9760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:09:03.223048 9760 webserver.cc:460] Webserver started at http://127.22.236.62:37867/ using document root <none> and password file <none>
I20260925 15:09:03.223690 9760 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:03.223785 9760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:03.224087 9760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:03.226879 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "96b4d1f690c840e186a9c6011294922a"
format_stamp: "Formatted at 2026-09-25 15:09:03 on dist-test-slave-nsrc"
server_key: "63eb508d91e7f42c56a33312e26eae59"
server_key_iv: "5c6602b1418ea29fa8acd7212bb3b095"
server_key_version: "encryptionkey@0"
I20260925 15:09:03.227638 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "96b4d1f690c840e186a9c6011294922a"
format_stamp: "Formatted at 2026-09-25 15:09:03 on dist-test-slave-nsrc"
server_key: "63eb508d91e7f42c56a33312e26eae59"
server_key_iv: "5c6602b1418ea29fa8acd7212bb3b095"
server_key_version: "encryptionkey@0"
I20260925 15:09:03.232470 9760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:09:03.235651 9774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:03.236912 9760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:03.237061 9760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "96b4d1f690c840e186a9c6011294922a"
format_stamp: "Formatted at 2026-09-25 15:09:03 on dist-test-slave-nsrc"
server_key: "63eb508d91e7f42c56a33312e26eae59"
server_key_iv: "5c6602b1418ea29fa8acd7212bb3b095"
server_key_version: "encryptionkey@0"
I20260925 15:09:03.237247 9760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:03.253938 9760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:03.254771 9760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:03.255029 9760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:03.271674 9760 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:35087
I20260925 15:09:03.271688 9825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:35087 every 8 connection(s)
I20260925 15:09:03.273005 9760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:03.276677 9826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:03.281581 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 9760
I20260925 15:09:03.281764 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:03.282308 23472 external_mini_cluster.cc:1428] Setting key 49c17aa7bbcdde067c891938c8448473
I20260925 15:09:03.287663 9826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a: Bootstrap starting.
I20260925 15:09:03.291105 9826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:03.292238 9826 log.cc:824] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:03.295266 9826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a: No bootstrap required, opened a new log
I20260925 15:09:03.296002 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:03.284174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.22.236.62:35087)
0925 15:09:03.285183 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:03.285194 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:03.285239 (+ 45us) server_negotiation.cc:407] Connection header received
0925 15:09:03.286942 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:03.286964 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:03.287419 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:03.287733 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:03.288353 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:03.289656 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:03.291536 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:03.292245 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:03.292397 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:03.293596 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:03.293604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:03.293618 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:03.294148 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:03.294222 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:03.294243 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:03.294335 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:03.294633 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 15:09:03.294960 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":341,"threads_started":1}
I20260925 15:09:03.302834 9826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b4d1f690c840e186a9c6011294922a" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35087 } }
I20260925 15:09:03.303220 9826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:03.303310 9826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b4d1f690c840e186a9c6011294922a, State: Initialized, Role: FOLLOWER
I20260925 15:09:03.303869 9826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b4d1f690c840e186a9c6011294922a" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35087 } }
I20260925 15:09:03.304107 9826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:03.304261 9826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:03.304417 9826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:03.306628 9826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b4d1f690c840e186a9c6011294922a" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35087 } }
I20260925 15:09:03.307126 9826 leader_election.cc:302] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96b4d1f690c840e186a9c6011294922a; no voters:
I20260925 15:09:03.307646 9826 leader_election.cc:288] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:03.307782 9831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:03.308763 9831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [term 1 LEADER]: Becoming Leader. State: Replica: 96b4d1f690c840e186a9c6011294922a, State: Running, Role: LEADER
I20260925 15:09:03.309391 9831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b4d1f690c840e186a9c6011294922a" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35087 } }
I20260925 15:09:03.309734 9826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:03.312029 9832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96b4d1f690c840e186a9c6011294922a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b4d1f690c840e186a9c6011294922a" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35087 } } }
I20260925 15:09:03.312444 9832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:03.312958 9833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96b4d1f690c840e186a9c6011294922a. Latest consensus state: current_term: 1 leader_uuid: "96b4d1f690c840e186a9c6011294922a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b4d1f690c840e186a9c6011294922a" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35087 } } }
I20260925 15:09:03.313233 9833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:03.313812 9840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:03.318578 9840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:03.327379 9840 catalog_manager.cc:1347] Generated new cluster ID: 8616a0e7ee864bb4a905ba7e2fc123ad
I20260925 15:09:03.327500 9840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:03.341078 9840 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:03.342767 9840 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:03.356850 9840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a: Generated new TSK 0
I20260925 15:09:03.357828 9840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:03.371469 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:35087
--builtin_ntp_servers=127.22.236.20:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:09:03.718191 9850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:03.718458 9850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:03.718578 9850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:03.725765 9850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:03.725871 9850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:03.726014 9850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:09:03.735394 9850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:35087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:03.736641 9850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:03.738307 9850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:03.750677 9856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:03.750770 9855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:03.750770 9858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:03.751333 9850 server_base.cc:1034] running on GCE node
I20260925 15:09:03.751937 9850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:03.752866 9850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:03.754076 9850 hybrid_clock.cc:648] HybridClock initialized: now 1790348943754038 us; error 29 us; skew 500 ppm
I20260925 15:09:03.754366 9850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:09:03.756551 9850 webserver.cc:460] Webserver started at http://127.22.236.1:42661/ using document root <none> and password file <none>
I20260925 15:09:03.757223 9850 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:03.757311 9850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:03.757598 9850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:03.760433 9850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "c9f3a10916b94e71bab6a92f7294eca9"
format_stamp: "Formatted at 2026-09-25 15:09:03 on dist-test-slave-nsrc"
server_key: "32ddbfa564c67d128a4e569e549b767b"
server_key_iv: "6ff91fa55b8a42a190e1d081af61b8df"
server_key_version: "encryptionkey@0"
I20260925 15:09:03.761162 9850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "c9f3a10916b94e71bab6a92f7294eca9"
format_stamp: "Formatted at 2026-09-25 15:09:03 on dist-test-slave-nsrc"
server_key: "32ddbfa564c67d128a4e569e549b767b"
server_key_iv: "6ff91fa55b8a42a190e1d081af61b8df"
server_key_version: "encryptionkey@0"
I20260925 15:09:03.766266 9850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:09:03.769572 9864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:03.770841 9850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:09:03.770982 9850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "c9f3a10916b94e71bab6a92f7294eca9"
format_stamp: "Formatted at 2026-09-25 15:09:03 on dist-test-slave-nsrc"
server_key: "32ddbfa564c67d128a4e569e549b767b"
server_key_iv: "6ff91fa55b8a42a190e1d081af61b8df"
server_key_version: "encryptionkey@0"
I20260925 15:09:03.771163 9850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:03.796862 9850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:03.797683 9850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:03.797966 9850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:03.798910 9850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:03.800347 9850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:03.800433 9850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:03.800540 9850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:03.800616 9850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:03.829773 9850 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:38031
I20260925 15:09:03.829833 9976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:38031 every 8 connection(s)
I20260925 15:09:03.831022 9850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:09:03.831265 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 9850
I20260925 15:09:03.831413 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:09:03.831864 23472 external_mini_cluster.cc:1428] Setting key 18f7958f4eec5738a0647cb47eb15c51
I20260925 15:09:03.835000 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:35087
--builtin_ntp_servers=127.22.236.20:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:09:03.844784 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:03.833499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:38609 (local address 127.22.236.62:35087)
0925 15:09:03.833925 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:03.833931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:03.835349 (+ 1418us) server_negotiation.cc:407] Connection header received
0925 15:09:03.837689 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:03.837694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:03.837758 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:03.837847 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:03.839087 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:03.839758 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:03.841856 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:03.842363 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:03.842473 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:03.843927 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:03.843932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:03.843935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:03.843954 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:03.844000 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:03.844012 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:03.844083 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:03.844327 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 15:09:03.844432 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":218,"threads_started":1}
I20260925 15:09:03.845745 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:03.834161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:35087 (local address 127.22.236.1:38609)
0925 15:09:03.835200 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:03.835245 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:03.837414 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:03.838090 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:03.838101 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:03.838213 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:03.838938 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:03.838962 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:03.839899 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:03.839912 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:03.841721 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:03.841735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:03.842578 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:03.842584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:03.842858 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:03.843629 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:09:03.843632 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:03.843675 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:09:03.843753 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:03.844098 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:03.844110 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:03.844117 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:09:03.844312 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 15:09:03.844688 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":275,"threads_started":1}
I20260925 15:09:03.847158 9977 heartbeater.cc:347] Connected to a master server at 127.22.236.62:35087
I20260925 15:09:03.847430 9977 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:03.848114 9977 heartbeater.cc:511] Master 127.22.236.62:35087 requested a full tablet report, sending...
I20260925 15:09:03.850204 9791 ts_manager.cc:194] Registered new tserver with Master: c9f3a10916b94e71bab6a92f7294eca9 (127.22.236.1:38031)
I20260925 15:09:03.852267 9791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.236.1:38609
W20260925 15:09:04.145628 9981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:04.145886 9981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:04.145987 9981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:04.153012 9981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:04.153122 9981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:04.153241 9981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:09:04.162096 9981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:35087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:04.163283 9981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:04.165028 9981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:04.177347 9986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:04.177306 9987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:04.177376 9989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:04.178128 9981 server_base.cc:1034] running on GCE node
I20260925 15:09:04.178675 9981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:04.179553 9981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:04.180744 9981 hybrid_clock.cc:648] HybridClock initialized: now 1790348944180703 us; error 29 us; skew 500 ppm
I20260925 15:09:04.181041 9981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:09:04.183153 9981 webserver.cc:460] Webserver started at http://127.22.236.2:36805/ using document root <none> and password file <none>
I20260925 15:09:04.183789 9981 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:04.183872 9981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:04.184131 9981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:04.186959 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "d8578f44c11a45e3a0780b287b580b49"
format_stamp: "Formatted at 2026-09-25 15:09:04 on dist-test-slave-nsrc"
server_key: "ad67cc235fe76ce8dbf57505940dcf98"
server_key_iv: "aebc8e67741daed6348c65aebc312c5a"
server_key_version: "encryptionkey@0"
I20260925 15:09:04.187668 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "d8578f44c11a45e3a0780b287b580b49"
format_stamp: "Formatted at 2026-09-25 15:09:04 on dist-test-slave-nsrc"
server_key: "ad67cc235fe76ce8dbf57505940dcf98"
server_key_iv: "aebc8e67741daed6348c65aebc312c5a"
server_key_version: "encryptionkey@0"
I20260925 15:09:04.192715 9981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260925 15:09:04.196185 9995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:04.197463 9981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:09:04.197606 9981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "d8578f44c11a45e3a0780b287b580b49"
format_stamp: "Formatted at 2026-09-25 15:09:04 on dist-test-slave-nsrc"
server_key: "ad67cc235fe76ce8dbf57505940dcf98"
server_key_iv: "aebc8e67741daed6348c65aebc312c5a"
server_key_version: "encryptionkey@0"
I20260925 15:09:04.197801 9981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:04.215646 9981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:04.216457 9981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:04.216719 9981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:04.217595 9981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:04.218940 9981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:04.219033 9981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:04.219134 9981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:04.219201 9981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:04.246009 9981 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:40035
I20260925 15:09:04.246057 10107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:40035 every 8 connection(s)
I20260925 15:09:04.247247 9981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:09:04.248948 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 9981
I20260925 15:09:04.249094 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:09:04.249552 23472 external_mini_cluster.cc:1428] Setting key 874de60975cd46c2f1df5f2fbe27e5b2
I20260925 15:09:04.252619 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:35087
--builtin_ntp_servers=127.22.236.20:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:09:04.260286 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:04.249657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:34501 (local address 127.22.236.62:35087)
0925 15:09:04.249847 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:04.249854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:04.251210 (+ 1356us) server_negotiation.cc:407] Connection header received
0925 15:09:04.253531 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:04.253536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:04.253603 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:04.253689 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:04.254984 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.255670 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.257672 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.258053 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.258141 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:04.259618 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:04.259623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:04.259626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:04.259645 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:04.259677 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:04.259688 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:04.259747 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:04.259944 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 15:09:04.260019 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:09:04.261451 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:04.250217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:35087 (local address 127.22.236.2:34501)
0925 15:09:04.251065 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:04.251116 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:04.253233 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:04.253939 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:04.253950 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:04.254071 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:04.254834 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:04.254857 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.255805 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.255810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:04.257552 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:04.257567 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.258249 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.258256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:04.258519 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:04.259271 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:09:04.259274 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:04.259322 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:09:04.259432 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:04.259759 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:04.259767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:04.259774 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:09:04.259936 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 15:09:04.260345 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":269,"threads_started":1}
I20260925 15:09:04.262843 10108 heartbeater.cc:347] Connected to a master server at 127.22.236.62:35087
I20260925 15:09:04.263106 10108 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:04.263811 10108 heartbeater.cc:511] Master 127.22.236.62:35087 requested a full tablet report, sending...
I20260925 15:09:04.265232 9791 ts_manager.cc:194] Registered new tserver with Master: d8578f44c11a45e3a0780b287b580b49 (127.22.236.2:40035)
I20260925 15:09:04.266247 9791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.236.2:34501
W20260925 15:09:04.570475 10111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:04.570753 10111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:04.570849 10111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:04.578125 10111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:04.578224 10111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:04.578329 10111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:09:04.587355 10111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:35087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:04.588574 10111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:04.590196 10111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:04.602489 10116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:04.602447 10119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:04.602414 10117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:04.603258 10111 server_base.cc:1034] running on GCE node
I20260925 15:09:04.603822 10111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:04.604775 10111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:04.605969 10111 hybrid_clock.cc:648] HybridClock initialized: now 1790348944605920 us; error 40 us; skew 500 ppm
I20260925 15:09:04.606268 10111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:09:04.608394 10111 webserver.cc:460] Webserver started at http://127.22.236.3:46785/ using document root <none> and password file <none>
I20260925 15:09:04.609040 10111 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:04.609136 10111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:04.609452 10111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:04.612376 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "f59aa1bb5a2943bb8b9517bc5f76272a"
format_stamp: "Formatted at 2026-09-25 15:09:04 on dist-test-slave-nsrc"
server_key: "e5495a9503546518f76d422351d41395"
server_key_iv: "e0ca9b9e3b9eb220e6d15b03b0b377f0"
server_key_version: "encryptionkey@0"
I20260925 15:09:04.613111 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "f59aa1bb5a2943bb8b9517bc5f76272a"
format_stamp: "Formatted at 2026-09-25 15:09:04 on dist-test-slave-nsrc"
server_key: "e5495a9503546518f76d422351d41395"
server_key_iv: "e0ca9b9e3b9eb220e6d15b03b0b377f0"
server_key_version: "encryptionkey@0"
I20260925 15:09:04.617879 10111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:09:04.621098 10125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:04.622298 10111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:09:04.622452 10111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "f59aa1bb5a2943bb8b9517bc5f76272a"
format_stamp: "Formatted at 2026-09-25 15:09:04 on dist-test-slave-nsrc"
server_key: "e5495a9503546518f76d422351d41395"
server_key_iv: "e0ca9b9e3b9eb220e6d15b03b0b377f0"
server_key_version: "encryptionkey@0"
I20260925 15:09:04.622637 10111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:04.634419 10111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:04.635170 10111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:04.635428 10111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:04.636286 10111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:04.637614 10111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:04.637699 10111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:04.637804 10111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:04.637869 10111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:04.666749 10111 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:34975
I20260925 15:09:04.666805 10237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:34975 every 8 connection(s)
I20260925 15:09:04.667999 10111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:09:04.677506 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 10111
I20260925 15:09:04.677663 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:09:04.678074 23472 external_mini_cluster.cc:1428] Setting key cf6370bf297e4f32dd4768097bfe39bf
I20260925 15:09:04.679461 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:04.670314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:43793 (local address 127.22.236.62:35087)
0925 15:09:04.670512 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:04.670518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:04.671880 (+ 1362us) server_negotiation.cc:407] Connection header received
0925 15:09:04.673541 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:04.673545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:04.673610 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:04.673694 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:04.674708 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.675465 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.676948 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.677332 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.677439 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:04.678764 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:04.678769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:04.678773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:04.678790 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:04.678822 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:04.678834 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:04.678923 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:04.679125 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 15:09:04.679214 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:09:04.680507 10240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:04.670881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:35087 (local address 127.22.236.3:43793)
0925 15:09:04.671740 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:04.671774 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:04.673313 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:04.673863 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:04.673871 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:04.674004 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:04.674573 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:04.674591 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.675600 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.675604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:04.676842 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:04.676854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.677543 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.677551 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:04.677804 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:04.678458 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:09:04.678460 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:04.678503 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:09:04.678573 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:04.678930 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:04.678939 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:04.678945 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:09:04.679113 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 15:09:04.679463 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":257,"threads_started":1}
I20260925 15:09:04.681684 10238 heartbeater.cc:347] Connected to a master server at 127.22.236.62:35087
I20260925 15:09:04.681936 10238 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:04.682585 10238 heartbeater.cc:511] Master 127.22.236.62:35087 requested a full tablet report, sending...
I20260925 15:09:04.683789 9791 ts_manager.cc:194] Registered new tserver with Master: f59aa1bb5a2943bb8b9517bc5f76272a (127.22.236.3:34975)
I20260925 15:09:04.684602 9791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.236.3:43793
I20260925 15:09:04.692863 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:09:04.702226 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:04.696755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.22.236.62:35087)
0925 15:09:04.696955 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:04.696961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:04.697322 (+ 361us) server_negotiation.cc:407] Connection header received
0925 15:09:04.697459 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:04.697463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:04.697540 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:04.697615 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:04.697997 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.698667 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.699975 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.700445 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.700554 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:04.701568 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:04.701573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:04.701577 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:04.701595 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:04.701626 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:04.701638 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:04.701822 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:04.701931 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 15:09:04.702004 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:09:04.705952 9791 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:09:04.712903 9791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:09:04.745285 10251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:04.724327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:34975 (local address 127.0.0.1:52382)
0925 15:09:04.736010 (+ 11683us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:04.736063 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:04.736289 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:04.737519 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:04.737526 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:04.737553 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:04.737987 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:04.738002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.739823 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.739829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:04.742280 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:04.742292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.744292 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.744299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:04.744492 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:04.744540 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:09:04.744671 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 15:09:04.744864 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11380,"mutex_wait_us":5,"thread_start_us":13902,"threads_started":1}
I20260925 15:09:04.745916 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:04.724544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:40035 (local address 127.0.0.1:53088)
0925 15:09:04.738494 (+ 13950us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:04.738511 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:04.738630 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:04.739490 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:04.739495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:04.739519 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:04.739938 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:04.739954 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.741783 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.741789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:04.743496 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:04.743517 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.745153 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.745160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:04.745352 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:04.745382 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:09:04.745471 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:09:04.745586 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13837,"mutex_wait_us":66,"thread_start_us":11934,"threads_started":1}
I20260925 15:09:04.745960 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:04.724635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.22.236.3:34975)
0925 15:09:04.736598 (+ 11963us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:04.736606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:04.736641 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:09:04.736742 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:04.736749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:04.736995 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:04.737191 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:04.738203 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.739707 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.742445 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.743744 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.743872 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:04.744136 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:04.745450 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0925 15:09:04.745617 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11804,"thread_start_us":11679,"threads_started":1}
I20260925 15:09:04.746696 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:04.724825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.22.236.2:40035)
0925 15:09:04.735927 (+ 11102us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:04.735937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:04.738627 (+ 2690us) server_negotiation.cc:407] Connection header received
0925 15:09:04.738769 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:04.738776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:04.739005 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:04.739202 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:04.740183 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.741647 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.743673 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.744967 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.745086 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:04.745325 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:04.746189 (+ 864us) server_negotiation.cc:299] Negotiation successful
0925 15:09:04.746348 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10909,"thread_start_us":11111,"threads_started":1}
I20260925 15:09:04.747963 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:04.726712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38031 (local address 127.0.0.1:41852)
0925 15:09:04.739574 (+ 12862us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:04.739589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:04.739716 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:04.740600 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:04.740604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:04.740627 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:04.741132 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:04.741144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.742756 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.742761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:04.744997 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:04.745012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.747249 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.747255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:04.747437 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:04.747467 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:09:04.747540 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:09:04.747654 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12750,"mutex_wait_us":1,"thread_start_us":12628,"threads_started":1}
I20260925 15:09:04.748080 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:04.727246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41852 (local address 127.22.236.1:38031)
0925 15:09:04.739190 (+ 11944us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:04.739198 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:04.739692 (+ 494us) server_negotiation.cc:407] Connection header received
0925 15:09:04.739919 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:04.739925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:04.740121 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:04.740353 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:04.741302 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.742624 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.745211 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:04.747081 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:04.747195 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:04.747405 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:04.747601 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 15:09:04.747753 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11807,"thread_start_us":11703,"threads_started":1}
I20260925 15:09:04.749644 10173 tablet_service.cc:1467] Processing CreateTablet for tablet 37146d25e59846fbbba60d7579feb292 (DEFAULT_TABLE table=test-table [id=a25bbe9fb6524ccf89c14693d9231d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:04.750883 10173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37146d25e59846fbbba60d7579feb292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:04.751247 9912 tablet_service.cc:1467] Processing CreateTablet for tablet 37146d25e59846fbbba60d7579feb292 (DEFAULT_TABLE table=test-table [id=a25bbe9fb6524ccf89c14693d9231d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:04.752348 9912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37146d25e59846fbbba60d7579feb292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:04.754456 10043 tablet_service.cc:1467] Processing CreateTablet for tablet 37146d25e59846fbbba60d7579feb292 (DEFAULT_TABLE table=test-table [id=a25bbe9fb6524ccf89c14693d9231d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:04.755481 10043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37146d25e59846fbbba60d7579feb292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:04.765370 10256 tablet_bootstrap.cc:492] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9: Bootstrap starting.
I20260925 15:09:04.769351 10256 tablet_bootstrap.cc:654] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:04.770673 10256 log.cc:824] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:04.772017 10257 tablet_bootstrap.cc:492] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49: Bootstrap starting.
I20260925 15:09:04.773562 10258 tablet_bootstrap.cc:492] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a: Bootstrap starting.
I20260925 15:09:04.774012 10256 tablet_bootstrap.cc:492] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9: No bootstrap required, opened a new log
I20260925 15:09:04.774317 10256 ts_tablet_manager.cc:1397] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260925 15:09:04.775390 10257 tablet_bootstrap.cc:654] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:04.776595 10257 log.cc:824] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:04.777297 10258 tablet_bootstrap.cc:654] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:04.778309 10258 log.cc:824] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:04.779508 10257 tablet_bootstrap.cc:492] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49: No bootstrap required, opened a new log
I20260925 15:09:04.779809 10257 ts_tablet_manager.cc:1397] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260925 15:09:04.781273 10258 tablet_bootstrap.cc:492] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a: No bootstrap required, opened a new log
I20260925 15:09:04.781520 10258 ts_tablet_manager.cc:1397] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 15:09:04.782823 10256 raft_consensus.cc:348] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59aa1bb5a2943bb8b9517bc5f76272a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 34975 } } peers { permanent_uuid: "d8578f44c11a45e3a0780b287b580b49" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40035 } } peers { permanent_uuid: "c9f3a10916b94e71bab6a92f7294eca9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38031 } }
I20260925 15:09:04.783180 10256 raft_consensus.cc:374] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:04.783303 10256 raft_consensus.cc:729] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f3a10916b94e71bab6a92f7294eca9, State: Initialized, Role: FOLLOWER
I20260925 15:09:04.783890 10256 consensus_queue.cc:260] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59aa1bb5a2943bb8b9517bc5f76272a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 34975 } } peers { permanent_uuid: "d8578f44c11a45e3a0780b287b580b49" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40035 } } peers { permanent_uuid: "c9f3a10916b94e71bab6a92f7294eca9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38031 } }
I20260925 15:09:04.784935 9977 heartbeater.cc:503] Master 127.22.236.62:35087 was elected leader, sending a full tablet report...
I20260925 15:09:04.785563 10256 ts_tablet_manager.cc:1428] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9: Time spent starting tablet: real 0.011s user 0.008s sys 0.001s
I20260925 15:09:04.790459 10257 raft_consensus.cc:348] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59aa1bb5a2943bb8b9517bc5f76272a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 34975 } } peers { permanent_uuid: "d8578f44c11a45e3a0780b287b580b49" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40035 } } peers { permanent_uuid: "c9f3a10916b94e71bab6a92f7294eca9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38031 } }
I20260925 15:09:04.790912 10257 raft_consensus.cc:374] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:04.791031 10257 raft_consensus.cc:729] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8578f44c11a45e3a0780b287b580b49, State: Initialized, Role: FOLLOWER
I20260925 15:09:04.791618 10257 consensus_queue.cc:260] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59aa1bb5a2943bb8b9517bc5f76272a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 34975 } } peers { permanent_uuid: "d8578f44c11a45e3a0780b287b580b49" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40035 } } peers { permanent_uuid: "c9f3a10916b94e71bab6a92f7294eca9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38031 } }
I20260925 15:09:04.791540 10258 raft_consensus.cc:348] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59aa1bb5a2943bb8b9517bc5f76272a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 34975 } } peers { permanent_uuid: "d8578f44c11a45e3a0780b287b580b49" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40035 } } peers { permanent_uuid: "c9f3a10916b94e71bab6a92f7294eca9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38031 } }
I20260925 15:09:04.791944 10258 raft_consensus.cc:374] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:04.792048 10258 raft_consensus.cc:729] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59aa1bb5a2943bb8b9517bc5f76272a, State: Initialized, Role: FOLLOWER
I20260925 15:09:04.792721 10108 heartbeater.cc:503] Master 127.22.236.62:35087 was elected leader, sending a full tablet report...
I20260925 15:09:04.792636 10258 consensus_queue.cc:260] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59aa1bb5a2943bb8b9517bc5f76272a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 34975 } } peers { permanent_uuid: "d8578f44c11a45e3a0780b287b580b49" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40035 } } peers { permanent_uuid: "c9f3a10916b94e71bab6a92f7294eca9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38031 } }
I20260925 15:09:04.793442 10257 ts_tablet_manager.cc:1428] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260925 15:09:04.793798 10238 heartbeater.cc:503] Master 127.22.236.62:35087 was elected leader, sending a full tablet report...
I20260925 15:09:04.794523 10258 ts_tablet_manager.cc:1428] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260925 15:09:04.833179 9978 tablet.cc:2369] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:09:04.919557 10239 tablet.cc:2369] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:09:04.999090 10109 tablet.cc:2369] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:09:05.144429 10263 raft_consensus.cc:482] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:09:05.144760 10263 raft_consensus.cc:504] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59aa1bb5a2943bb8b9517bc5f76272a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 34975 } } peers { permanent_uuid: "d8578f44c11a45e3a0780b287b580b49" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40035 } } peers { permanent_uuid: "c9f3a10916b94e71bab6a92f7294eca9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38031 } }
I20260925 15:09:05.146477 10263 leader_election.cc:288] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f59aa1bb5a2943bb8b9517bc5f76272a (127.22.236.3:34975), c9f3a10916b94e71bab6a92f7294eca9 (127.22.236.1:38031)
I20260925 15:09:05.152897 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.146925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:57407 (local address 127.22.236.1:38031)
0925 15:09:05.147118 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:05.147124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:05.147825 (+ 701us) server_negotiation.cc:407] Connection header received
0925 15:09:05.148037 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:05.148041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:05.148118 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:05.148230 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:05.148831 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.149515 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.151269 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.152011 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.152183 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:05.152255 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:05.152527 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 15:09:05.152645 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:09:05.152912 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.147002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:34283 (local address 127.22.236.3:34975)
0925 15:09:05.147159 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:05.147165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:05.148060 (+ 895us) server_negotiation.cc:407] Connection header received
0925 15:09:05.148217 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:05.148222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:05.148281 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:05.148351 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:05.148963 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.149648 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.151333 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.152066 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.152230 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:05.152298 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:05.152557 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 15:09:05.152678 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:09:05.153019 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.146903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38031 (local address 127.22.236.2:57407)
0925 15:09:05.147684 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:05.147704 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:05.147895 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:05.148274 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:05.148279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:05.148303 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:05.148686 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:05.148696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.149641 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.149645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:05.151093 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:05.151108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.152215 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.152222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:05.152373 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:05.152395 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:09:05.152480 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 15:09:05.152603 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":1,"spinlock_wait_cycles":24320,"thread_start_us":744,"threads_started":1}
I20260925 15:09:05.153076 10265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.146911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:34975 (local address 127.22.236.2:34283)
0925 15:09:05.147973 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:05.147988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:05.148087 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:05.148404 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:05.148409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:05.148456 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:05.148814 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:05.148822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.149775 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.149783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:05.151157 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:05.151169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.152253 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.152259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:05.152416 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:05.152438 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:09:05.152512 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:09:05.152627 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":28,"spinlock_wait_cycles":22272,"thread_start_us":392,"threads_started":1}
I20260925 15:09:05.153991 10193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37146d25e59846fbbba60d7579feb292" candidate_uuid: "d8578f44c11a45e3a0780b287b580b49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59aa1bb5a2943bb8b9517bc5f76272a" is_pre_election: true
I20260925 15:09:05.153991 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37146d25e59846fbbba60d7579feb292" candidate_uuid: "d8578f44c11a45e3a0780b287b580b49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f3a10916b94e71bab6a92f7294eca9" is_pre_election: true
I20260925 15:09:05.154430 9932 raft_consensus.cc:2454] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8578f44c11a45e3a0780b287b580b49 in term 0.
I20260925 15:09:05.154433 10193 raft_consensus.cc:2454] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8578f44c11a45e3a0780b287b580b49 in term 0.
I20260925 15:09:05.155022 9996 leader_election.cc:302] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8578f44c11a45e3a0780b287b580b49, f59aa1bb5a2943bb8b9517bc5f76272a; no voters:
I20260925 15:09:05.155369 10263 raft_consensus.cc:2781] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:09:05.155503 10263 raft_consensus.cc:482] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:09:05.155614 10263 raft_consensus.cc:3037] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:05.157732 10263 raft_consensus.cc:504] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59aa1bb5a2943bb8b9517bc5f76272a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 34975 } } peers { permanent_uuid: "d8578f44c11a45e3a0780b287b580b49" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40035 } } peers { permanent_uuid: "c9f3a10916b94e71bab6a92f7294eca9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38031 } }
I20260925 15:09:05.158268 10263 leader_election.cc:288] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [CANDIDATE]: Term 1 election: Requested vote from peers f59aa1bb5a2943bb8b9517bc5f76272a (127.22.236.3:34975), c9f3a10916b94e71bab6a92f7294eca9 (127.22.236.1:38031)
I20260925 15:09:05.158699 10193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37146d25e59846fbbba60d7579feb292" candidate_uuid: "d8578f44c11a45e3a0780b287b580b49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59aa1bb5a2943bb8b9517bc5f76272a"
I20260925 15:09:05.158754 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37146d25e59846fbbba60d7579feb292" candidate_uuid: "d8578f44c11a45e3a0780b287b580b49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f3a10916b94e71bab6a92f7294eca9"
I20260925 15:09:05.158898 10193 raft_consensus.cc:3037] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:05.158950 9932 raft_consensus.cc:3037] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:05.162006 9932 raft_consensus.cc:2454] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8578f44c11a45e3a0780b287b580b49 in term 1.
I20260925 15:09:05.162006 10193 raft_consensus.cc:2454] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8578f44c11a45e3a0780b287b580b49 in term 1.
I20260925 15:09:05.162436 9998 leader_election.cc:302] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f3a10916b94e71bab6a92f7294eca9, d8578f44c11a45e3a0780b287b580b49; no voters:
I20260925 15:09:05.162712 10263 raft_consensus.cc:2781] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:05.163128 10263 raft_consensus.cc:686] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [term 1 LEADER]: Becoming Leader. State: Replica: d8578f44c11a45e3a0780b287b580b49, State: Running, Role: LEADER
I20260925 15:09:05.163568 10263 consensus_queue.cc:237] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59aa1bb5a2943bb8b9517bc5f76272a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 34975 } } peers { permanent_uuid: "d8578f44c11a45e3a0780b287b580b49" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40035 } } peers { permanent_uuid: "c9f3a10916b94e71bab6a92f7294eca9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38031 } }
I20260925 15:09:05.167900 9791 catalog_manager.cc:5495] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8578f44c11a45e3a0780b287b580b49 (127.22.236.2). New cstate: current_term: 1 leader_uuid: "d8578f44c11a45e3a0780b287b580b49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59aa1bb5a2943bb8b9517bc5f76272a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8578f44c11a45e3a0780b287b580b49" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 40035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f3a10916b94e71bab6a92f7294eca9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38031 } health_report { overall_health: UNKNOWN } } }
I20260925 15:09:05.202625 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.197323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.22.236.2:40035)
0925 15:09:05.197503 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:05.197509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:05.197530 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:09:05.197683 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:05.197688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:05.197766 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:05.197892 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:05.198266 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.198976 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.200308 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.200802 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.200910 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:05.201509 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:05.202013 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:05.202172 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:05.202306 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 15:09:05.202348 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:09:05.210558 9932 raft_consensus.cc:1261] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9 [term 1 FOLLOWER]: Refusing update from remote peer d8578f44c11a45e3a0780b287b580b49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:05.210570 10193 raft_consensus.cc:1261] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a [term 1 FOLLOWER]: Refusing update from remote peer d8578f44c11a45e3a0780b287b580b49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:05.211472 10267 consensus_queue.cc:1035] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f59aa1bb5a2943bb8b9517bc5f76272a" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:05.211776 10263 consensus_queue.cc:1035] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f3a10916b94e71bab6a92f7294eca9" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:05.223340 10272 mvcc.cc:205] Tried to move back new op lower bound from 7333269279572398080 to 7333269279393665024. Current Snapshot: MvccSnapshot[applied={T|T < 7333269279572398080}]
I20260925 15:09:05.234396 9790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:09:05.234642 9790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:09:05.237974 9790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96b4d1f690c840e186a9c6011294922a: Sending DeleteTablet for 3 replicas of tablet 37146d25e59846fbbba60d7579feb292
I20260925 15:09:05.239214 10043 tablet_service.cc:1514] Processing DeleteTablet for tablet 37146d25e59846fbbba60d7579feb292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:05 UTC) from {username='slave'} at 127.0.0.1:53088
I20260925 15:09:05.239334 9912 tablet_service.cc:1514] Processing DeleteTablet for tablet 37146d25e59846fbbba60d7579feb292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:05 UTC) from {username='slave'} at 127.0.0.1:41852
I20260925 15:09:05.239723 10173 tablet_service.cc:1514] Processing DeleteTablet for tablet 37146d25e59846fbbba60d7579feb292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:05 UTC) from {username='slave'} at 127.0.0.1:52382
I20260925 15:09:05.241688 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 9850
I20260925 15:09:05.250787 10281 tablet_replica.cc:330] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a: stopping tablet replica
I20260925 15:09:05.251245 10281 raft_consensus.cc:2229] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:09:05.251775 10281 raft_consensus.cc:2258] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:05.252669 10280 tablet_replica.cc:330] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49: stopping tablet replica
I20260925 15:09:05.253190 10280 raft_consensus.cc:2229] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:09:05.253871 10280 raft_consensus.cc:2258] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:05.253963 10281 ts_tablet_manager.cc:1905] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:05.254578 10279 tablet_replica.cc:330] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9: stopping tablet replica
I20260925 15:09:05.256278 10280 ts_tablet_manager.cc:1905] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:05.261368 10281 ts_tablet_manager.cc:1918] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:09:05.261631 10281 log.cc:1196] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/37146d25e59846fbbba60d7579feb292
I20260925 15:09:05.262195 10281 ts_tablet_manager.cc:1939] T 37146d25e59846fbbba60d7579feb292 P f59aa1bb5a2943bb8b9517bc5f76272a: Deleting consensus metadata
I20260925 15:09:05.263363 10280 ts_tablet_manager.cc:1918] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:09:05.263541 10280 log.cc:1196] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/37146d25e59846fbbba60d7579feb292
I20260925 15:09:05.263986 10280 ts_tablet_manager.cc:1939] T 37146d25e59846fbbba60d7579feb292 P d8578f44c11a45e3a0780b287b580b49: Deleting consensus metadata
I20260925 15:09:05.264302 9775 catalog_manager.cc:4841] TS f59aa1bb5a2943bb8b9517bc5f76272a (127.22.236.3:34975): tablet 37146d25e59846fbbba60d7579feb292 (table test-table [id=a25bbe9fb6524ccf89c14693d9231d21]) successfully deleted
I20260925 15:09:05.262801 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.243180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.22.236.1:38031)
0925 15:09:05.243375 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:05.243381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:05.248379 (+ 4998us) server_negotiation.cc:407] Connection header received
0925 15:09:05.248465 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:05.248471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:05.248617 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:05.248729 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:05.251026 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.252301 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.258581 (+ 6280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.259259 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.259377 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:05.261073 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:05.261084 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:05.261101 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:05.261882 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:05.261941 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:05.261967 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:05.262053 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:05.262314 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 15:09:05.262434 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:09:05.265527 9778 catalog_manager.cc:4841] TS d8578f44c11a45e3a0780b287b580b49 (127.22.236.2:40035): tablet 37146d25e59846fbbba60d7579feb292 (table test-table [id=a25bbe9fb6524ccf89c14693d9231d21]) successfully deleted
I20260925 15:09:05.265970 10279 raft_consensus.cc:2229] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:09:05.266341 9972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:05.372584 10279 raft_consensus.cc:2258] T 37146d25e59846fbbba60d7579feb292 P c9f3a10916b94e71bab6a92f7294eca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:05.373447 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 9850
W20260925 15:09:05.394960 9777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.236.1:38031 (error 108)
I20260925 15:09:05.395174 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 9981
I20260925 15:09:05.397994 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.396670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38031 (local address 127.0.0.1:41870)
0925 15:09:05.397639 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:05.397774 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":735,"threads_started":1}
W20260925 15:09:05.398382 9777 catalog_manager.cc:4584] TS c9f3a10916b94e71bab6a92f7294eca9 (127.22.236.1:38031): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37146d25e59846fbbba60d7579feb292: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38031: connect: Connection refused (error 111)
I20260925 15:09:05.403470 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.396611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.22.236.2:40035)
0925 15:09:05.396889 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:05.396895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:05.396917 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:09:05.397014 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:05.397049 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:05.397190 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:05.397834 (+ 644us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:05.397987 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.398955 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.400349 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.400854 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.400996 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:05.402110 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:05.402120 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:05.402135 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:05.402743 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:05.402790 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:05.402811 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:05.402915 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:05.403052 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:09:05.403159 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 15:09:05.404124 10103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:05.452220 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.451647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38031 (local address 127.0.0.1:41872)
0925 15:09:05.451889 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:05.452009 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 15:09:05.453285 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.452882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38031 (local address 127.0.0.1:41880)
0925 15:09:05.453033 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:05.453137 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 15:09:05.513718 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.513221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38031 (local address 127.0.0.1:41896)
0925 15:09:05.513414 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:05.513528 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:09:05.514807 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.514408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38031 (local address 127.0.0.1:41910)
0925 15:09:05.514555 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:05.514635 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 15:09:05.518270 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 9981
I20260925 15:09:05.532785 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 10111
I20260925 15:09:05.538975 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.533483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52394 (local address 127.22.236.3:34975)
0925 15:09:05.533661 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:05.533666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:05.533686 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:09:05.533855 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:05.533868 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:05.533950 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:05.534036 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:05.534394 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.535111 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.536363 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.536764 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.536866 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:05.537752 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:05.537763 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:05.537783 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:05.538335 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:05.538382 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:05.538402 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:05.538483 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:05.538555 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 15:09:05.538642 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:09:05.539636 10233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:05.595461 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.594872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38031 (local address 127.0.0.1:41916)
0925 15:09:05.595157 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:05.595275 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 15:09:05.596553 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.596080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:38031 (local address 127.0.0.1:41924)
0925 15:09:05.596296 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:05.596378 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:38031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260925 15:09:05.627354 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 10111
I20260925 15:09:05.639946 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 9760
I20260925 15:09:05.641050 9821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:05.747749 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 9760
2026-09-25T15:09:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2942 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:05 dist-test-slave-nsrc krb5kdc[10291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:05 dist-test-slave-nsrc krb5kdc[10291](info): set up 2 sockets
Sep 25 15:09:05 dist-test-slave-nsrc krb5kdc[10291](info): commencing operation
krb5kdc: starting...
W20260925 15:09:06.748234 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:06 dist-test-slave-nsrc krb5kdc[10291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:09:06.975131 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:34711
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:45457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:34711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:07.294651 10304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:07.295007 10304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:07.295149 10304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:07.302711 10304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:07.302834 10304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:07.302907 10304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:07.302984 10304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:07.303045 10304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:07.312911 10304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:45457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:34711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:34711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:07.314214 10304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:07.315486 10304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:07.326279 10309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:07.326231 10310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:07.326314 10312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:07.327116 10304 server_base.cc:1034] running on GCE node
I20260925 15:09:07.327850 10304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:07.329180 10304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:07.330420 10304 hybrid_clock.cc:648] HybridClock initialized: now 1790348947330374 us; error 28 us; skew 500 ppm
I20260925 15:09:07.330740 10304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:07 dist-test-slave-nsrc krb5kdc[10291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348947, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:07.337319 10304 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:09:07.338744 10304 webserver.cc:460] Webserver started at http://127.22.236.62:39157/ using document root <none> and password file <none>
I20260925 15:09:07.339375 10304 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:07.339476 10304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:07.339716 10304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:07.342617 10304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "4c4963bdfde34dbe9ca4d7716c871441"
format_stamp: "Formatted at 2026-09-25 15:09:07 on dist-test-slave-nsrc"
server_key: "7f5599354df12def9076c34b6156c143"
server_key_iv: "82abab45ceb8b6ac7ab40f891f8d935e"
server_key_version: "encryptionkey@0"
I20260925 15:09:07.343315 10304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "4c4963bdfde34dbe9ca4d7716c871441"
format_stamp: "Formatted at 2026-09-25 15:09:07 on dist-test-slave-nsrc"
server_key: "7f5599354df12def9076c34b6156c143"
server_key_iv: "82abab45ceb8b6ac7ab40f891f8d935e"
server_key_version: "encryptionkey@0"
I20260925 15:09:07.348172 10304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 15:09:07.351339 10319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:07.352615 10304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 15:09:07.352751 10304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "4c4963bdfde34dbe9ca4d7716c871441"
format_stamp: "Formatted at 2026-09-25 15:09:07 on dist-test-slave-nsrc"
server_key: "7f5599354df12def9076c34b6156c143"
server_key_iv: "82abab45ceb8b6ac7ab40f891f8d935e"
server_key_version: "encryptionkey@0"
I20260925 15:09:07.352912 10304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:07.368985 10304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:07.373027 10304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:07.373252 10304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:07.389477 10304 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:34711
I20260925 15:09:07.389501 10370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:34711 every 8 connection(s)
I20260925 15:09:07.390829 10304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:07.394598 10371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:07.396654 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 10304
I20260925 15:09:07.396817 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:07.397222 23472 external_mini_cluster.cc:1428] Setting key 557fb31f67db07c5ba5ce9614b7ceb69
I20260925 15:09:07.408740 10371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441: Bootstrap starting.
Sep 25 15:09:07 dist-test-slave-nsrc krb5kdc[10291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:09:07.411844 10371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:07.413070 10371 log.cc:824] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:07.415968 10371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441: No bootstrap required, opened a new log
I20260925 15:09:07.419728 10374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:07.398965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.22.236.62:34711)
0925 15:09:07.399727 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:07.399748 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:07.399808 (+ 60us) server_negotiation.cc:407] Connection header received
0925 15:09:07.401559 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:07.401590 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:07.402135 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:07.402563 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:07.403876 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.405935 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.407235 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.408312 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.408492 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:07.412428 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:07.412439 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:07.412457 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:07.412499 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:07.415563 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:07.416439 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:07.416445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:07.416451 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:07.416633 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:07.417055 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:07.417058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:07.417060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:07.417608 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:07.417828 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:07.418297 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 15:09:07.418685 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":295,"threads_started":1}
I20260925 15:09:07.424571 10371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4963bdfde34dbe9ca4d7716c871441" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 34711 } }
I20260925 15:09:07.424934 10371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:07.425019 10371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c4963bdfde34dbe9ca4d7716c871441, State: Initialized, Role: FOLLOWER
I20260925 15:09:07.425609 10371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4963bdfde34dbe9ca4d7716c871441" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 34711 } }
I20260925 15:09:07.425869 10371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:07.425999 10371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:07.426157 10371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:07.428267 10371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4963bdfde34dbe9ca4d7716c871441" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 34711 } }
I20260925 15:09:07.428730 10371 leader_election.cc:302] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c4963bdfde34dbe9ca4d7716c871441; no voters:
I20260925 15:09:07.429323 10371 leader_election.cc:288] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:07.429435 10376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:07.430016 10376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [term 1 LEADER]: Becoming Leader. State: Replica: 4c4963bdfde34dbe9ca4d7716c871441, State: Running, Role: LEADER
I20260925 15:09:07.430586 10376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4963bdfde34dbe9ca4d7716c871441" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 34711 } }
I20260925 15:09:07.431412 10371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:07.433274 10378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c4963bdfde34dbe9ca4d7716c871441. Latest consensus state: current_term: 1 leader_uuid: "4c4963bdfde34dbe9ca4d7716c871441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4963bdfde34dbe9ca4d7716c871441" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 34711 } } }
I20260925 15:09:07.433301 10377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c4963bdfde34dbe9ca4d7716c871441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4963bdfde34dbe9ca4d7716c871441" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 34711 } } }
I20260925 15:09:07.433835 10378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:07.433835 10377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:07.434751 10385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:07.447932 10385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:07.456823 10385 catalog_manager.cc:1347] Generated new cluster ID: 5b231497901145ca867fefb4a39189f1
I20260925 15:09:07.456951 10385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:07.475464 10385 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:07.477352 10385 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:07.484802 10385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441: Generated new TSK 0
I20260925 15:09:07.485569 10385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.236.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
Entry for principal kudu/127.22.236.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab.
I20260925 15:09:07.554911 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:34711
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:45457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:07.882897 10397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:07.883165 10397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:07.883297 10397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:07.890534 10397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:07.890647 10397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:07.890802 10397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:09:07.900866 10397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:45457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.1.keytab
--principal=kudu/127.22.236.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:34711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:07.902118 10397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:07.903359 10397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:07.915712 10402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:07.915795 10403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:07.915732 10405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:07.916579 10397 server_base.cc:1034] running on GCE node
I20260925 15:09:07.917098 10397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:07.917963 10397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:07.919149 10397 hybrid_clock.cc:648] HybridClock initialized: now 1790348947919107 us; error 38 us; skew 500 ppm
I20260925 15:09:07.919473 10397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:07 dist-test-slave-nsrc krb5kdc[10291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348947, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:07.922580 10397 init.cc:377] Logged in from keytab as kudu/127.22.236.1@KRBTEST.COM (short username kudu)
I20260925 15:09:07.923940 10397 webserver.cc:460] Webserver started at http://127.22.236.1:35777/ using document root <none> and password file <none>
I20260925 15:09:07.924624 10397 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:07.924734 10397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:07.924995 10397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:07.927676 10397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "b75f9dad7bba407db2e28f18f1cc2878"
format_stamp: "Formatted at 2026-09-25 15:09:07 on dist-test-slave-nsrc"
server_key: "2d06d5bdbede77a12e89d2ac02fd83f0"
server_key_iv: "83ef1ee42bbfbc8b2888d8f28da4576a"
server_key_version: "encryptionkey@0"
I20260925 15:09:07.928416 10397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "b75f9dad7bba407db2e28f18f1cc2878"
format_stamp: "Formatted at 2026-09-25 15:09:07 on dist-test-slave-nsrc"
server_key: "2d06d5bdbede77a12e89d2ac02fd83f0"
server_key_iv: "83ef1ee42bbfbc8b2888d8f28da4576a"
server_key_version: "encryptionkey@0"
I20260925 15:09:07.933316 10397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:09:07.936566 10412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:07.937739 10397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 15:09:07.937889 10397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "b75f9dad7bba407db2e28f18f1cc2878"
format_stamp: "Formatted at 2026-09-25 15:09:07 on dist-test-slave-nsrc"
server_key: "2d06d5bdbede77a12e89d2ac02fd83f0"
server_key_iv: "83ef1ee42bbfbc8b2888d8f28da4576a"
server_key_version: "encryptionkey@0"
I20260925 15:09:07.938051 10397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:07.970508 10397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:07.974579 10397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:07.974833 10397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:07.975708 10397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:07.977029 10397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:07.977099 10397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:07.977212 10397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:07.977263 10397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:08.006393 10397 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:33937
I20260925 15:09:08.006469 10524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:33937 every 8 connection(s)
I20260925 15:09:08.007635 10397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:09:08.009552 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 10397
I20260925 15:09:08.009697 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:09:08.010177 23472 external_mini_cluster.cc:1428] Setting key 072cff9794f45d8b04a3f88628d7a9da
Sep 25 15:09:08 dist-test-slave-nsrc krb5kdc[10291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348947, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.1@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:09:08.027608 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:08.010184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:55755 (local address 127.22.236.62:34711)
0925 15:09:08.010678 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:08.010685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:08.011749 (+ 1064us) server_negotiation.cc:407] Connection header received
0925 15:09:08.013625 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:08.013631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:08.013712 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:08.013822 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:08.015250 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.016208 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.017819 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.018358 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.018482 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:08.022091 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:08.022096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:08.022100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:08.022135 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:08.024611 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:08.025453 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:08.025456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:08.025459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:08.025585 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:08.026032 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:08.026035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:08.026036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:08.026287 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:08.026391 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:08.027190 (+ 799us) server_negotiation.cc:299] Negotiation successful
0925 15:09:08.027360 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":233,"threads_started":1}
I20260925 15:09:08.029029 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:08.010762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:34711 (local address 127.22.236.1:55755)
0925 15:09:08.011589 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:08.011634 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:08.013357 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:08.014005 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:08.014013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:08.014544 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:08.015093 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:08.015111 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.016352 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.016355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:08.017681 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:08.017693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.018560 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.018564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:08.018758 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:08.019374 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:08.019401 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:08.021859 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:08.024758 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:08.024767 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:08.024787 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:08.025352 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:08.025681 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:08.025686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:08.025689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:08.025944 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:08.026395 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:08.026402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:08.026834 (+ 432us) client_negotiation.cc:769] Sending connection context
0925 15:09:08.027135 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 15:09:08.027638 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":333,"threads_started":1}
I20260925 15:09:08.030579 10525 heartbeater.cc:347] Connected to a master server at 127.22.236.62:34711
I20260925 15:09:08.030912 10525 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:08.031836 10525 heartbeater.cc:511] Master 127.22.236.62:34711 requested a full tablet report, sending...
I20260925 15:09:08.034029 10336 ts_manager.cc:194] Registered new tserver with Master: b75f9dad7bba407db2e28f18f1cc2878 (127.22.236.1:33937)
WARNING: no policy specified for kudu/127.22.236.2@KRBTEST.COM; defaulting to no policy
I20260925 15:09:08.035595 10336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.1@KRBTEST.COM'} at 127.22.236.1:55755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
Entry for principal kudu/127.22.236.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab.
I20260925 15:09:08.062453 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:34711
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:45457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:08.367025 10531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:08.367296 10531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:08.367417 10531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:08.374470 10531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:08.374586 10531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:08.374745 10531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:09:08.384289 10531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:45457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.2.keytab
--principal=kudu/127.22.236.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:34711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:08.385540 10531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:08.386793 10531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:08.399453 10536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:08.399473 10537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:08.399863 10539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:08.401443 10531 server_base.cc:1034] running on GCE node
I20260925 15:09:08.402144 10531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:08.403321 10531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:08.404556 10531 hybrid_clock.cc:648] HybridClock initialized: now 1790348948404498 us; error 45 us; skew 500 ppm
I20260925 15:09:08.404930 10531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:08 dist-test-slave-nsrc krb5kdc[10291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348948, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:08.407995 10531 init.cc:377] Logged in from keytab as kudu/127.22.236.2@KRBTEST.COM (short username kudu)
I20260925 15:09:08.409338 10531 webserver.cc:460] Webserver started at http://127.22.236.2:44329/ using document root <none> and password file <none>
I20260925 15:09:08.409987 10531 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:08.410101 10531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:08.410387 10531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:08.413163 10531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "7925134af73247f68e4bf0dbbed97705"
format_stamp: "Formatted at 2026-09-25 15:09:08 on dist-test-slave-nsrc"
server_key: "c77558aa556345dbe273b475478985a1"
server_key_iv: "468815330e9c25a2a241e27113992464"
server_key_version: "encryptionkey@0"
I20260925 15:09:08.413841 10531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "7925134af73247f68e4bf0dbbed97705"
format_stamp: "Formatted at 2026-09-25 15:09:08 on dist-test-slave-nsrc"
server_key: "c77558aa556345dbe273b475478985a1"
server_key_iv: "468815330e9c25a2a241e27113992464"
server_key_version: "encryptionkey@0"
I20260925 15:09:08.418660 10531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:09:08.421886 10546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:08.423070 10531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:09:08.423206 10531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "7925134af73247f68e4bf0dbbed97705"
format_stamp: "Formatted at 2026-09-25 15:09:08 on dist-test-slave-nsrc"
server_key: "c77558aa556345dbe273b475478985a1"
server_key_iv: "468815330e9c25a2a241e27113992464"
server_key_version: "encryptionkey@0"
I20260925 15:09:08.423373 10531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:08.459482 10531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:08.463240 10531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:08.463522 10531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:08.464496 10531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:08.465783 10531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:08.465857 10531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:08.465975 10531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:08.466027 10531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:08.494830 10531 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:37781
I20260925 15:09:08.494908 10658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:37781 every 8 connection(s)
I20260925 15:09:08.496031 10531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:09:08.501771 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 10531
I20260925 15:09:08.501932 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:09:08.502378 23472 external_mini_cluster.cc:1428] Setting key ed5f72807f496ff1c8599e5f6da3af8b
Sep 25 15:09:08 dist-test-slave-nsrc krb5kdc[10291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348948, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.2@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:09:08.515803 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:08.498395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:60353 (local address 127.22.236.62:34711)
0925 15:09:08.498552 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:08.498557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:08.499668 (+ 1111us) server_negotiation.cc:407] Connection header received
0925 15:09:08.501502 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:08.501509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:08.501589 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:08.501675 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:08.503348 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.504357 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.505925 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.506491 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.506608 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:08.510185 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:08.510190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:08.510194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:08.510227 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:08.513041 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:08.513712 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:08.513717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:08.513721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:08.513871 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:08.514217 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:08.514221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:08.514225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:08.514545 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:08.514680 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:08.515276 (+ 596us) server_negotiation.cc:299] Negotiation successful
0925 15:09:08.515488 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:09:08.516688 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:08.498843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:34711 (local address 127.22.236.2:60353)
0925 15:09:08.499505 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:08.499541 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:08.501200 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:08.501918 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:08.501929 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:08.502459 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:08.503155 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:08.503184 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.504513 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.504517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:08.505761 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:08.505774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.506670 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.506675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:08.506871 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:08.507498 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:08.507519 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:08.509970 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:08.513170 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:08.513177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:08.513190 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:08.513608 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:08.513963 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:08.513966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:08.513968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:08.514134 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:08.514647 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:08.514652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:08.514948 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 15:09:08.515165 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 15:09:08.515566 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":260,"threads_started":1}
I20260925 15:09:08.518066 10659 heartbeater.cc:347] Connected to a master server at 127.22.236.62:34711
I20260925 15:09:08.518343 10659 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:08.519006 10659 heartbeater.cc:511] Master 127.22.236.62:34711 requested a full tablet report, sending...
I20260925 15:09:08.520572 10336 ts_manager.cc:194] Registered new tserver with Master: 7925134af73247f68e4bf0dbbed97705 (127.22.236.2:37781)
I20260925 15:09:08.521347 10336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.2@KRBTEST.COM'} at 127.22.236.2:60353
WARNING: no policy specified for kudu/127.22.236.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
Entry for principal kudu/127.22.236.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab.
I20260925 15:09:08.553730 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:34711
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:45457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:08.864125 10664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:08.864426 10664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:08.864571 10664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:08.871529 10664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:08.871640 10664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:08.871796 10664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:09:08.881654 10664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:45457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.3.keytab
--principal=kudu/127.22.236.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:34711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:08.883028 10664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:08.884378 10664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:08.896797 10670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:08.896942 10672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:08.896800 10669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:08.897604 10664 server_base.cc:1034] running on GCE node
I20260925 15:09:08.898145 10664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:08.899013 10664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:08.900219 10664 hybrid_clock.cc:648] HybridClock initialized: now 1790348948900173 us; error 37 us; skew 500 ppm
I20260925 15:09:08.900512 10664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:08 dist-test-slave-nsrc krb5kdc[10291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348948, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:08.903434 10664 init.cc:377] Logged in from keytab as kudu/127.22.236.3@KRBTEST.COM (short username kudu)
I20260925 15:09:08.905364 10664 webserver.cc:460] Webserver started at http://127.22.236.3:45839/ using document root <none> and password file <none>
I20260925 15:09:08.906028 10664 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:08.906141 10664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:08.906425 10664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:08.909232 10664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c"
format_stamp: "Formatted at 2026-09-25 15:09:08 on dist-test-slave-nsrc"
server_key: "567ee49482e4581a52532375b5ee05ba"
server_key_iv: "e1b5e2d0f280b68cfc3cf1bfcfb193e0"
server_key_version: "encryptionkey@0"
I20260925 15:09:08.909945 10664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c"
format_stamp: "Formatted at 2026-09-25 15:09:08 on dist-test-slave-nsrc"
server_key: "567ee49482e4581a52532375b5ee05ba"
server_key_iv: "e1b5e2d0f280b68cfc3cf1bfcfb193e0"
server_key_version: "encryptionkey@0"
I20260925 15:09:08.914911 10664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:09:08.918222 10679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:08.919469 10664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:09:08.919613 10664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c"
format_stamp: "Formatted at 2026-09-25 15:09:08 on dist-test-slave-nsrc"
server_key: "567ee49482e4581a52532375b5ee05ba"
server_key_iv: "e1b5e2d0f280b68cfc3cf1bfcfb193e0"
server_key_version: "encryptionkey@0"
I20260925 15:09:08.919775 10664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:08.946157 10664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:08.950379 10664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:08.950651 10664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:08.951570 10664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:08.952844 10664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:08.952922 10664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:08.953037 10664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:08.953089 10664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:08.981916 10664 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:44711
I20260925 15:09:08.981966 10791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:44711 every 8 connection(s)
I20260925 15:09:08.983103 10664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:09:08.984498 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 10664
I20260925 15:09:08.984668 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:09:08.985126 23472 external_mini_cluster.cc:1428] Setting key 7c54cebea8ce72307879095f9fc42f90
Sep 25 15:09:08 dist-test-slave-nsrc krb5kdc[10291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348948, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.3@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:09:09.002069 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:08.985636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:47053 (local address 127.22.236.62:34711)
0925 15:09:08.985836 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:08.985841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:08.987086 (+ 1245us) server_negotiation.cc:407] Connection header received
0925 15:09:08.988955 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:08.988959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:08.989045 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:08.989124 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:08.990472 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.991196 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.992711 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.993090 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.993176 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:08.996443 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:08.996448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:08.996451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:08.996483 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:08.999036 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:08.999785 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:08.999789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:08.999791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:08.999922 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:09.000370 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:09.000374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:09.000376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:09.000661 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:09.000798 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:09.001492 (+ 694us) server_negotiation.cc:299] Negotiation successful
0925 15:09:09.001714 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:09:09.003031 10794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:08.986143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:34711 (local address 127.22.236.3:47053)
0925 15:09:08.986947 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:08.986989 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:08.988722 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:08.989313 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:08.989322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:08.989810 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:08.990350 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:08.990367 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.991322 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.991325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:08.992584 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:08.992600 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.993263 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.993268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:08.993457 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:08.994042 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:08.994063 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:08.996243 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:08.999193 (+ 2950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:08.999200 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:08.999213 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:08.999628 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:09.000026 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:09.000029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:09.000031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:09.000261 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:09.000783 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:09.000791 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:09.001128 (+ 337us) client_negotiation.cc:769] Sending connection context
0925 15:09:09.001373 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 15:09:09.001892 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":323,"threads_started":1}
I20260925 15:09:09.004381 10792 heartbeater.cc:347] Connected to a master server at 127.22.236.62:34711
I20260925 15:09:09.004662 10792 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:09.005383 10792 heartbeater.cc:511] Master 127.22.236.62:34711 requested a full tablet report, sending...
I20260925 15:09:09.006932 10336 ts_manager.cc:194] Registered new tserver with Master: 9fabbc61ddb540e5a2d347f44eb5ee3c (127.22.236.3:44711)
I20260925 15:09:09.008040 10336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.236.3@KRBTEST.COM'} at 127.22.236.3:47053
I20260925 15:09:09.011996 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:09:09.027678 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.015772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49536 (local address 127.22.236.62:34711)
0925 15:09:09.015930 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:09.015936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:09.016379 (+ 443us) server_negotiation.cc:407] Connection header received
0925 15:09:09.016550 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:09.016555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:09.016620 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:09.016694 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:09.017996 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.018779 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.020013 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.020472 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.020582 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.022743 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:09.022748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:09.022752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:09.022781 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:09.025157 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:09.025768 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:09.025773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:09.025777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:09.025907 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:09.026283 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:09.026289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:09.026292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:09.026622 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:09.026866 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:09.027161 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 15:09:09.027365 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:09:09.031108 10336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:09:09.037853 10336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:09:09.038426 10525 heartbeater.cc:503] Master 127.22.236.62:34711 was elected leader, sending a full tablet report...
I20260925 15:09:09.070467 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.049676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.22.236.2:37781)
0925 15:09:09.061679 (+ 12003us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:09.061689 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:09.062745 (+ 1056us) server_negotiation.cc:407] Connection header received
0925 15:09:09.062849 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:09.062854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:09.063085 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:09.063258 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:09.064302 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.065853 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.068102 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.069343 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.069467 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.069673 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:09.069976 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 15:09:09.070142 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11822,"thread_start_us":11683,"threads_started":1}
I20260925 15:09:09.070565 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.049331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:37781 (local address 127.0.0.1:36646)
0925 15:09:09.062622 (+ 13291us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:09.062638 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:09.062739 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:09.063523 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:09.063532 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:09.063580 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:09.064089 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:09.064100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.065995 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.066007 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:09.067929 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:09.067949 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.069545 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.069552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:09.069757 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.069797 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 15:09:09.069923 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:09:09.070105 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13154,"mutex_wait_us":113,"spinlock_wait_cycles":1792,"thread_start_us":12514,"threads_started":1}
I20260925 15:09:09.071377 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.049225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33937 (local address 127.0.0.1:40028)
0925 15:09:09.062236 (+ 13011us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:09.062287 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:09.062524 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:09.063533 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:09.063539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:09.063593 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:09.064573 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:09.064591 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.066297 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.066302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:09.068491 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:09.068504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.070577 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.070584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:09.070738 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.070772 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:09:09.070845 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:09:09.070979 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12749,"mutex_wait_us":476,"spinlock_wait_cycles":11648,"thread_start_us":13108,"threads_started":1}
I20260925 15:09:09.072083 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.049326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40028 (local address 127.22.236.1:33937)
0925 15:09:09.062015 (+ 12689us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:09.062023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:09.062635 (+ 612us) server_negotiation.cc:407] Connection header received
0925 15:09:09.062757 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:09.062765 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:09.063041 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:09.063235 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:09.064811 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.066180 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.068665 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.070084 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.070213 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.070428 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:09.071606 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0925 15:09:09.071773 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12526,"thread_start_us":12709,"threads_started":1}
I20260925 15:09:09.072853 10808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.063064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:44711 (local address 127.0.0.1:44708)
0925 15:09:09.064876 (+ 1812us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:09.064891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:09.064992 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:09.065859 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:09.065862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:09.065880 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:09.066268 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:09.066283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.068239 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.068244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:09.070245 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:09.070262 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.072063 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.072070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:09.072254 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.072289 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:09:09.072381 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:09:09.072498 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1705,"mutex_wait_us":1,"thread_start_us":1552,"threads_started":1}
I20260925 15:09:09.073148 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.063153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.22.236.3:44711)
0925 15:09:09.064414 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:09.064423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:09.065096 (+ 673us) server_negotiation.cc:407] Connection header received
0925 15:09:09.065195 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:09.065202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:09.065439 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:09.065608 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:09.066497 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.068057 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.070428 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.071773 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.071889 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.072087 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:09.072576 (+ 489us) server_negotiation.cc:299] Negotiation successful
0925 15:09:09.072725 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1092,"thread_start_us":892,"threads_started":1}
I20260925 15:09:09.074546 10594 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd2d49e36ce46a8aa61406ced89db9c (DEFAULT_TABLE table=test-table [id=356e3e87fb864e7a8755e880e7c62c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:09.076004 10594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd2d49e36ce46a8aa61406ced89db9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:09.076296 10727 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd2d49e36ce46a8aa61406ced89db9c (DEFAULT_TABLE table=test-table [id=356e3e87fb864e7a8755e880e7c62c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:09.076264 10460 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd2d49e36ce46a8aa61406ced89db9c (DEFAULT_TABLE table=test-table [id=356e3e87fb864e7a8755e880e7c62c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:09.077580 10727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd2d49e36ce46a8aa61406ced89db9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:09.077641 10460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd2d49e36ce46a8aa61406ced89db9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:09.093428 10811 tablet_bootstrap.cc:492] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705: Bootstrap starting.
I20260925 15:09:09.094332 10810 tablet_bootstrap.cc:492] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c: Bootstrap starting.
I20260925 15:09:09.096175 10812 tablet_bootstrap.cc:492] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878: Bootstrap starting.
I20260925 15:09:09.098455 10811 tablet_bootstrap.cc:654] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:09.098455 10810 tablet_bootstrap.cc:654] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:09.099350 10812 tablet_bootstrap.cc:654] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:09.099428 10811 log.cc:824] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:09.099593 10810 log.cc:824] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:09.100461 10812 log.cc:824] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:09.102133 10810 tablet_bootstrap.cc:492] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c: No bootstrap required, opened a new log
I20260925 15:09:09.102361 10810 ts_tablet_manager.cc:1397] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 15:09:09.102463 10811 tablet_bootstrap.cc:492] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705: No bootstrap required, opened a new log
I20260925 15:09:09.102733 10811 ts_tablet_manager.cc:1397] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260925 15:09:09.106621 10812 tablet_bootstrap.cc:492] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878: No bootstrap required, opened a new log
I20260925 15:09:09.106891 10812 ts_tablet_manager.cc:1397] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260925 15:09:09.110225 10810 raft_consensus.cc:348] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75f9dad7bba407db2e28f18f1cc2878" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33937 } } peers { permanent_uuid: "7925134af73247f68e4bf0dbbed97705" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 37781 } } peers { permanent_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44711 } }
I20260925 15:09:09.110582 10810 raft_consensus.cc:374] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:09.110688 10810 raft_consensus.cc:729] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fabbc61ddb540e5a2d347f44eb5ee3c, State: Initialized, Role: FOLLOWER
I20260925 15:09:09.111254 10810 consensus_queue.cc:260] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75f9dad7bba407db2e28f18f1cc2878" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33937 } } peers { permanent_uuid: "7925134af73247f68e4bf0dbbed97705" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 37781 } } peers { permanent_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44711 } }
I20260925 15:09:09.112342 10792 heartbeater.cc:503] Master 127.22.236.62:34711 was elected leader, sending a full tablet report...
I20260925 15:09:09.112582 10811 raft_consensus.cc:348] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75f9dad7bba407db2e28f18f1cc2878" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33937 } } peers { permanent_uuid: "7925134af73247f68e4bf0dbbed97705" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 37781 } } peers { permanent_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44711 } }
I20260925 15:09:09.112998 10811 raft_consensus.cc:374] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:09.113085 10811 raft_consensus.cc:729] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7925134af73247f68e4bf0dbbed97705, State: Initialized, Role: FOLLOWER
I20260925 15:09:09.113341 10810 ts_tablet_manager.cc:1428] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260925 15:09:09.113785 10811 consensus_queue.cc:260] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75f9dad7bba407db2e28f18f1cc2878" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33937 } } peers { permanent_uuid: "7925134af73247f68e4bf0dbbed97705" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 37781 } } peers { permanent_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44711 } }
I20260925 15:09:09.114874 10659 heartbeater.cc:503] Master 127.22.236.62:34711 was elected leader, sending a full tablet report...
I20260925 15:09:09.115489 10811 ts_tablet_manager.cc:1428] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260925 15:09:09.118207 10812 raft_consensus.cc:348] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75f9dad7bba407db2e28f18f1cc2878" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33937 } } peers { permanent_uuid: "7925134af73247f68e4bf0dbbed97705" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 37781 } } peers { permanent_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44711 } }
I20260925 15:09:09.118645 10812 raft_consensus.cc:374] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:09.118731 10812 raft_consensus.cc:729] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75f9dad7bba407db2e28f18f1cc2878, State: Initialized, Role: FOLLOWER
I20260925 15:09:09.119241 10812 consensus_queue.cc:260] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75f9dad7bba407db2e28f18f1cc2878" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33937 } } peers { permanent_uuid: "7925134af73247f68e4bf0dbbed97705" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 37781 } } peers { permanent_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44711 } }
I20260925 15:09:09.120664 10812 ts_tablet_manager.cc:1428] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260925 15:09:09.234674 10793 tablet.cc:2369] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:09:09.247881 10660 tablet.cc:2369] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:09:09.259807 10526 tablet.cc:2369] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:09:09.457247 10817 raft_consensus.cc:482] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:09:09.457561 10817 raft_consensus.cc:504] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75f9dad7bba407db2e28f18f1cc2878" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33937 } } peers { permanent_uuid: "7925134af73247f68e4bf0dbbed97705" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 37781 } } peers { permanent_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44711 } }
I20260925 15:09:09.457829 10818 raft_consensus.cc:482] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:09:09.458091 10818 raft_consensus.cc:504] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75f9dad7bba407db2e28f18f1cc2878" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33937 } } peers { permanent_uuid: "7925134af73247f68e4bf0dbbed97705" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 37781 } } peers { permanent_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44711 } }
I20260925 15:09:09.459709 10817 leader_election.cc:288] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b75f9dad7bba407db2e28f18f1cc2878 (127.22.236.1:33937), 9fabbc61ddb540e5a2d347f44eb5ee3c (127.22.236.3:44711)
I20260925 15:09:09.460426 10818 leader_election.cc:288] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7925134af73247f68e4bf0dbbed97705 (127.22.236.2:37781), 9fabbc61ddb540e5a2d347f44eb5ee3c (127.22.236.3:44711)
I20260925 15:09:09.469010 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.461511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:51915 (local address 127.22.236.3:44711)
0925 15:09:09.461702 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:09.461707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:09.461829 (+ 122us) server_negotiation.cc:407] Connection header received
0925 15:09:09.462117 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:09.462122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:09.462203 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:09.462264 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:09.463112 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.464197 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.466638 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.467821 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.467949 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.468021 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:09.468600 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 15:09:09.468731 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:09:09.469875 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.460914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:44711 (local address 127.22.236.1:51915)
0925 15:09:09.461729 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:09.461745 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:09.461946 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:09.462472 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:09.462477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:09.462505 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:09.462937 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:09.462948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.464366 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.464374 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:09.466468 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:09.466483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.468230 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.468237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:09.468411 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.468445 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:09:09.469352 (+ 907us) client_negotiation.cc:240] Negotiation successful
0925 15:09:09.469483 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":3,"thread_start_us":773,"threads_started":1}
I20260925 15:09:09.470180 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.461159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:34151 (local address 127.22.236.1:33937)
0925 15:09:09.461319 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:09.461324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:09.462602 (+ 1278us) server_negotiation.cc:407] Connection header received
0925 15:09:09.462893 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:09.462899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:09.463000 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:09.463087 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:09.464184 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.465380 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.467803 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.468967 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.469081 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.469143 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:09.469719 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 15:09:09.469845 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:09:09.470373 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.459986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:33937 (local address 127.22.236.2:34151)
0925 15:09:09.462510 (+ 2524us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:09.462534 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:09.462719 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:09.463354 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:09.463359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:09.463423 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:09.463946 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:09.463960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.465561 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.465566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:09.467620 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:09.467636 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.469345 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.469351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:09.469539 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.469574 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:09:09.469909 (+ 335us) client_negotiation.cc:240] Negotiation successful
0925 15:09:09.470047 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2330,"thread_start_us":2094,"threads_started":1}
I20260925 15:09:09.472081 10747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd2d49e36ce46a8aa61406ced89db9c" candidate_uuid: "b75f9dad7bba407db2e28f18f1cc2878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" is_pre_election: true
I20260925 15:09:09.472069 10480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd2d49e36ce46a8aa61406ced89db9c" candidate_uuid: "7925134af73247f68e4bf0dbbed97705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75f9dad7bba407db2e28f18f1cc2878" is_pre_election: true
I20260925 15:09:09.472539 10480 raft_consensus.cc:2454] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7925134af73247f68e4bf0dbbed97705 in term 0.
I20260925 15:09:09.472618 10747 raft_consensus.cc:2454] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b75f9dad7bba407db2e28f18f1cc2878 in term 0.
I20260925 15:09:09.473253 10547 leader_election.cc:302] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7925134af73247f68e4bf0dbbed97705, b75f9dad7bba407db2e28f18f1cc2878; no voters:
I20260925 15:09:09.473234 10413 leader_election.cc:302] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fabbc61ddb540e5a2d347f44eb5ee3c, b75f9dad7bba407db2e28f18f1cc2878; no voters:
I20260925 15:09:09.473690 10818 raft_consensus.cc:2781] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:09:09.473742 10817 raft_consensus.cc:2781] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:09:09.473865 10817 raft_consensus.cc:482] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:09:09.473870 10818 raft_consensus.cc:482] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:09:09.473969 10817 raft_consensus.cc:3037] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:09.473989 10818 raft_consensus.cc:3037] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:09.477164 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.462904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:55379 (local address 127.22.236.3:44711)
0925 15:09:09.469240 (+ 6336us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:09.469244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:09.470786 (+ 1542us) server_negotiation.cc:407] Connection header received
0925 15:09:09.470925 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:09.470931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:09.471020 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:09.471087 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:09.471509 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.472687 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.474907 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.476017 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.476164 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.476234 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:09.476754 (+ 520us) server_negotiation.cc:299] Negotiation successful
0925 15:09:09.476886 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6243,"thread_start_us":7262,"threads_started":1}
I20260925 15:09:09.477087 10818 raft_consensus.cc:504] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75f9dad7bba407db2e28f18f1cc2878" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33937 } } peers { permanent_uuid: "7925134af73247f68e4bf0dbbed97705" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 37781 } } peers { permanent_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44711 } }
I20260925 15:09:09.477082 10817 raft_consensus.cc:504] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75f9dad7bba407db2e28f18f1cc2878" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33937 } } peers { permanent_uuid: "7925134af73247f68e4bf0dbbed97705" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 37781 } } peers { permanent_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44711 } }
I20260925 15:09:09.477685 10820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.460077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:37781 (local address 127.22.236.1:39729)
0925 15:09:09.461185 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:09.461204 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:09.461346 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:09.471239 (+ 9893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:09.471243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:09.471269 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:09.471616 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:09.471629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.473051 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.473056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:09.475133 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:09.475147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.476947 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.476953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:09.477129 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.477162 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:09:09.477253 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:09:09.477376 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":740,"threads_started":1}
I20260925 15:09:09.477751 10817 leader_election.cc:288] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [CANDIDATE]: Term 1 election: Requested vote from peers b75f9dad7bba407db2e28f18f1cc2878 (127.22.236.1:33937), 9fabbc61ddb540e5a2d347f44eb5ee3c (127.22.236.3:44711)
I20260925 15:09:09.477850 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.462575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:44711 (local address 127.22.236.2:55379)
0925 15:09:09.470690 (+ 8115us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:09.470707 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:09.470815 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:09.471117 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:09.471121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:09.471144 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:09.471358 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:09.471369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.472844 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.472849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:09.474724 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:09.474757 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.476373 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.476379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:09.476593 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.476623 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:09:09.477365 (+ 742us) client_negotiation.cc:240] Negotiation successful
0925 15:09:09.477524 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8019,"spinlock_wait_cycles":25856,"thread_start_us":7930,"threads_started":1}
I20260925 15:09:09.478217 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.470745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:39729 (local address 127.22.236.2:37781)
0925 15:09:09.470906 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:09.470912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:09.470927 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:09:09.470985 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:09.470990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:09.471058 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:09.471145 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:09.471736 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.472787 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.475297 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.476602 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.476728 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.476786 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:09.477860 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0925 15:09:09.477996 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:09:09.478310 10480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd2d49e36ce46a8aa61406ced89db9c" candidate_uuid: "7925134af73247f68e4bf0dbbed97705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75f9dad7bba407db2e28f18f1cc2878"
I20260925 15:09:09.478606 10480 raft_consensus.cc:2379] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7925134af73247f68e4bf0dbbed97705 in current term 1: Already voted for candidate b75f9dad7bba407db2e28f18f1cc2878 in this term.
I20260925 15:09:09.478863 10746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd2d49e36ce46a8aa61406ced89db9c" candidate_uuid: "7925134af73247f68e4bf0dbbed97705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c"
I20260925 15:09:09.478953 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd2d49e36ce46a8aa61406ced89db9c" candidate_uuid: "b75f9dad7bba407db2e28f18f1cc2878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7925134af73247f68e4bf0dbbed97705" is_pre_election: true
I20260925 15:09:09.478614 10818 leader_election.cc:288] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [CANDIDATE]: Term 1 election: Requested vote from peers 7925134af73247f68e4bf0dbbed97705 (127.22.236.2:37781), 9fabbc61ddb540e5a2d347f44eb5ee3c (127.22.236.3:44711)
I20260925 15:09:09.479159 10746 raft_consensus.cc:3037] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:09.479125 10745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd2d49e36ce46a8aa61406ced89db9c" candidate_uuid: "b75f9dad7bba407db2e28f18f1cc2878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c"
I20260925 15:09:09.479401 10614 raft_consensus.cc:2379] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b75f9dad7bba407db2e28f18f1cc2878 in current term 1: Already voted for candidate 7925134af73247f68e4bf0dbbed97705 in this term.
I20260925 15:09:09.479446 10747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd2d49e36ce46a8aa61406ced89db9c" candidate_uuid: "7925134af73247f68e4bf0dbbed97705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" is_pre_election: true
I20260925 15:09:09.480099 10614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd2d49e36ce46a8aa61406ced89db9c" candidate_uuid: "b75f9dad7bba407db2e28f18f1cc2878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7925134af73247f68e4bf0dbbed97705"
I20260925 15:09:09.480322 10614 raft_consensus.cc:2379] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b75f9dad7bba407db2e28f18f1cc2878 in current term 1: Already voted for candidate 7925134af73247f68e4bf0dbbed97705 in this term.
I20260925 15:09:09.481062 10414 leader_election.cc:302] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b75f9dad7bba407db2e28f18f1cc2878; no voters: 7925134af73247f68e4bf0dbbed97705, 9fabbc61ddb540e5a2d347f44eb5ee3c
I20260925 15:09:09.481338 10818 raft_consensus.cc:2726] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 15:09:09.482137 10746 raft_consensus.cc:2454] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7925134af73247f68e4bf0dbbed97705 in term 1.
I20260925 15:09:09.482466 10547 leader_election.cc:302] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7925134af73247f68e4bf0dbbed97705, 9fabbc61ddb540e5a2d347f44eb5ee3c; no voters: b75f9dad7bba407db2e28f18f1cc2878
I20260925 15:09:09.482752 10817 raft_consensus.cc:2781] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:09.483152 10817 raft_consensus.cc:686] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 1 LEADER]: Becoming Leader. State: Replica: 7925134af73247f68e4bf0dbbed97705, State: Running, Role: LEADER
I20260925 15:09:09.483585 10817 consensus_queue.cc:237] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75f9dad7bba407db2e28f18f1cc2878" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33937 } } peers { permanent_uuid: "7925134af73247f68e4bf0dbbed97705" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 37781 } } peers { permanent_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44711 } }
I20260925 15:09:09.487803 10336 catalog_manager.cc:5495] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7925134af73247f68e4bf0dbbed97705 (127.22.236.2). New cstate: current_term: 1 leader_uuid: "7925134af73247f68e4bf0dbbed97705" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75f9dad7bba407db2e28f18f1cc2878" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7925134af73247f68e4bf0dbbed97705" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 37781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44711 } health_report { overall_health: UNKNOWN } } }
I20260925 15:09:09.529636 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.524264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.22.236.2:37781)
0925 15:09:09.524451 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:09.524457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:09.524476 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:09:09.524702 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:09.524706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:09.524813 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:09.524938 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:09.525287 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.526052 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.527438 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.527898 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.528008 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.528550 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:09.529056 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:09.529185 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:09.529280 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 15:09:09.529361 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:09:09.537438 10746 raft_consensus.cc:1261] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c [term 1 FOLLOWER]: Refusing update from remote peer 7925134af73247f68e4bf0dbbed97705: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:09.537452 10480 raft_consensus.cc:1261] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 1 FOLLOWER]: Refusing update from remote peer 7925134af73247f68e4bf0dbbed97705: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:09.538216 10824 consensus_queue.cc:1035] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fabbc61ddb540e5a2d347f44eb5ee3c" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 44711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:09.538534 10817 consensus_queue.cc:1035] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75f9dad7bba407db2e28f18f1cc2878" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:09.561954 10336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:09:09.562181 10336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:09:09.565332 10336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c4963bdfde34dbe9ca4d7716c871441: Sending DeleteTablet for 3 replicas of tablet 7bd2d49e36ce46a8aa61406ced89db9c
I20260925 15:09:09.566445 10594 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bd2d49e36ce46a8aa61406ced89db9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:09 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:36646
I20260925 15:09:09.566438 10460 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bd2d49e36ce46a8aa61406ced89db9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:09 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:40028
I20260925 15:09:09.566824 10727 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bd2d49e36ce46a8aa61406ced89db9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:09 UTC) from {username='kudu', principal='kudu/127.22.236.62@KRBTEST.COM'} at 127.0.0.1:44708
I20260925 15:09:09.568787 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 10397
I20260925 15:09:09.581918 10837 tablet_replica.cc:330] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878: stopping tablet replica
I20260925 15:09:09.582748 10839 tablet_replica.cc:330] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c: stopping tablet replica
I20260925 15:09:09.583226 10839 raft_consensus.cc:2229] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:09:09.583316 10837 raft_consensus.cc:2229] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:09:09.583751 10839 raft_consensus.cc:2258] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:09.584559 10837 raft_consensus.cc:2258] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:09.585417 10838 tablet_replica.cc:330] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705: stopping tablet replica
I20260925 15:09:09.585827 10838 raft_consensus.cc:2229] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:09:09.586145 10839 ts_tablet_manager.cc:1905] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:09.586846 10837 ts_tablet_manager.cc:1905] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:09.589949 10838 raft_consensus.cc:2258] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:09.593271 10839 ts_tablet_manager.cc:1918] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 25 15:09:09 dist-test-slave-nsrc krb5kdc[10291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.1@KRBTEST.COM
I20260925 15:09:09.593519 10839 log.cc:1196] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/7bd2d49e36ce46a8aa61406ced89db9c
I20260925 15:09:09.593972 10837 ts_tablet_manager.cc:1918] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:09:09.594206 10837 log.cc:1196] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-0/wal/wals/7bd2d49e36ce46a8aa61406ced89db9c
I20260925 15:09:09.594364 10839 ts_tablet_manager.cc:1939] T 7bd2d49e36ce46a8aa61406ced89db9c P 9fabbc61ddb540e5a2d347f44eb5ee3c: Deleting consensus metadata
I20260925 15:09:09.596374 10320 catalog_manager.cc:4841] TS 9fabbc61ddb540e5a2d347f44eb5ee3c (127.22.236.3:44711): tablet 7bd2d49e36ce46a8aa61406ced89db9c (table test-table [id=356e3e87fb864e7a8755e880e7c62c75]) successfully deleted
I20260925 15:09:09.595777 10837 ts_tablet_manager.cc:1939] T 7bd2d49e36ce46a8aa61406ced89db9c P b75f9dad7bba407db2e28f18f1cc2878: Deleting consensus metadata
I20260925 15:09:09.597047 10838 ts_tablet_manager.cc:1905] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:09.598276 10320 catalog_manager.cc:4841] TS b75f9dad7bba407db2e28f18f1cc2878 (127.22.236.1:33937): tablet 7bd2d49e36ce46a8aa61406ced89db9c (table test-table [id=356e3e87fb864e7a8755e880e7c62c75]) successfully deleted
I20260925 15:09:09.601754 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.570051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.22.236.1:33937)
0925 15:09:09.570223 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:09.570230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:09.582137 (+ 11907us) server_negotiation.cc:407] Connection header received
0925 15:09:09.582319 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:09.582325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:09.582425 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:09.582548 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:09.584602 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.585759 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.587576 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.588346 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.588462 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.594859 (+ 6397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:09.594872 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:09.594890 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:09.594930 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:09.598634 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:09.599440 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:09.599448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:09.599452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:09.599613 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:09.600027 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:09.600032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:09.600035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:09.600502 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:09.600797 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:09.601143 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 15:09:09.601386 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:09:09.602525 10520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:09.603911 10838 ts_tablet_manager.cc:1918] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:09:09.604120 10838 log.cc:1196] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/7bd2d49e36ce46a8aa61406ced89db9c
I20260925 15:09:09.604984 10838 ts_tablet_manager.cc:1939] T 7bd2d49e36ce46a8aa61406ced89db9c P 7925134af73247f68e4bf0dbbed97705: Deleting consensus metadata
I20260925 15:09:09.607088 10321 catalog_manager.cc:4841] TS 7925134af73247f68e4bf0dbbed97705 (127.22.236.2:37781): tablet 7bd2d49e36ce46a8aa61406ced89db9c (table test-table [id=356e3e87fb864e7a8755e880e7c62c75]) successfully deleted
I20260925 15:09:09.694324 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 10397
I20260925 15:09:09.708887 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 10531
Sep 25 15:09:09 dist-test-slave-nsrc krb5kdc[10291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.2@KRBTEST.COM
I20260925 15:09:09.724704 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.709655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.22.236.2:37781)
0925 15:09:09.709869 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:09.709875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:09.709896 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:09:09.709990 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:09.709995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:09.710081 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:09.710192 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:09.711346 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.712509 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.713761 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.714449 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.714571 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.718712 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:09.718726 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:09.718742 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:09.718779 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:09.721814 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:09.722653 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:09.722659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:09.722662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:09.722818 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:09.723200 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:09.723202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:09.723204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:09.723653 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:09.723898 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:09.724224 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 15:09:09.724425 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 15:09:09.725425 10654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:09.835522 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 10531
I20260925 15:09:09.850477 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 10664
Sep 25 15:09:09 dist-test-slave-nsrc krb5kdc[10291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.3@KRBTEST.COM
I20260925 15:09:09.865057 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:09.851165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.22.236.3:44711)
0925 15:09:09.851326 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:09.851331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:09.851367 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:09:09.851557 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:09.851561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:09.851647 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:09.851729 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:09.853043 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.853719 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.855008 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:09.855427 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:09.855534 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:09.859820 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:09.859835 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:09.859851 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:09.859887 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:09.862523 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:09.863152 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:09.863159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:09.863161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:09.863282 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:09.863614 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:09.863618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:09.863620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:09.863942 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:09.864346 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:09.864515 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 15:09:09.864767 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:09:09.865626 10787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:09.953230 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 10664
I20260925 15:09:09.965937 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 10304
I20260925 15:09:09.966938 10366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:10.073644 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 10304
2026-09-25T15:09:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4326 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T15:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:10Z Disabled control of system clock
I20260925 15:09:10.140654 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:36947
--webserver_interface=127.22.236.62
--webserver_port=0
--builtin_ntp_servers=127.22.236.20:45241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:36947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:09:10.454453 10848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:10.454747 10848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:10.454882 10848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:10.462080 10848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:10.462199 10848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:10.462265 10848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:10.462344 10848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:10.462404 10848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:10.471541 10848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:45241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:36947
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:36947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:10.472782 10848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:10.474498 10848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:10.485371 10856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:10.485373 10854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:10.485587 10853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:10.486187 10848 server_base.cc:1034] running on GCE node
I20260925 15:09:10.486915 10848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:10.488263 10848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:10.489445 10848 hybrid_clock.cc:648] HybridClock initialized: now 1790348950489386 us; error 54 us; skew 500 ppm
I20260925 15:09:10.489760 10848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:09:10.491889 10848 webserver.cc:460] Webserver started at http://127.22.236.62:40273/ using document root <none> and password file <none>
I20260925 15:09:10.492583 10848 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:10.492682 10848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:10.493003 10848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:10.495740 10848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "70039d6f48d4425eae48f2aa15e0b77d"
format_stamp: "Formatted at 2026-09-25 15:09:10 on dist-test-slave-nsrc"
server_key: "b845be5b9af9cdb5c2d8f0b1d2a824be"
server_key_iv: "64d17f6dd36ed8bc05723f7697581228"
server_key_version: "encryptionkey@0"
I20260925 15:09:10.496531 10848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "70039d6f48d4425eae48f2aa15e0b77d"
format_stamp: "Formatted at 2026-09-25 15:09:10 on dist-test-slave-nsrc"
server_key: "b845be5b9af9cdb5c2d8f0b1d2a824be"
server_key_iv: "64d17f6dd36ed8bc05723f7697581228"
server_key_version: "encryptionkey@0"
I20260925 15:09:10.501304 10848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:09:10.504535 10862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:10.505818 10848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:09:10.505972 10848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "70039d6f48d4425eae48f2aa15e0b77d"
format_stamp: "Formatted at 2026-09-25 15:09:10 on dist-test-slave-nsrc"
server_key: "b845be5b9af9cdb5c2d8f0b1d2a824be"
server_key_iv: "64d17f6dd36ed8bc05723f7697581228"
server_key_version: "encryptionkey@0"
I20260925 15:09:10.506161 10848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:10.521569 10848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:10.522343 10848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:10.522594 10848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:10.539218 10848 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:36947
I20260925 15:09:10.539238 10913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:36947 every 8 connection(s)
I20260925 15:09:10.540616 10848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:10.544268 10914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:10.550477 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 10848
I20260925 15:09:10.550635 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:10.551065 23472 external_mini_cluster.cc:1428] Setting key 926f9471b0d3e79fe8f2da9bf8820e94
I20260925 15:09:10.554423 10914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d: Bootstrap starting.
I20260925 15:09:10.557507 10914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:10.558508 10914 log.cc:824] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:10.561170 10914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d: No bootstrap required, opened a new log
I20260925 15:09:10.563864 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:10.552973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.22.236.62:36947)
0925 15:09:10.553880 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:10.553891 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:10.553931 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:09:10.555652 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:10.555675 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:10.556131 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:10.556532 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:10.557051 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:10.558340 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:10.559659 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:10.560357 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:10.560530 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:10.561635 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:10.561642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:10.561655 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:10.562176 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:10.562254 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:10.562275 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:10.562378 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:10.562598 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 15:09:10.562899 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":310,"threads_started":1}
I20260925 15:09:10.569285 10914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70039d6f48d4425eae48f2aa15e0b77d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36947 } }
I20260925 15:09:10.569682 10914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:10.569777 10914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70039d6f48d4425eae48f2aa15e0b77d, State: Initialized, Role: FOLLOWER
I20260925 15:09:10.570353 10914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70039d6f48d4425eae48f2aa15e0b77d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36947 } }
I20260925 15:09:10.570585 10914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:10.570708 10914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:10.570858 10914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:10.573284 10914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70039d6f48d4425eae48f2aa15e0b77d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36947 } }
I20260925 15:09:10.573770 10914 leader_election.cc:302] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70039d6f48d4425eae48f2aa15e0b77d; no voters:
I20260925 15:09:10.574342 10914 leader_election.cc:288] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:10.574491 10919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:10.575153 10919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [term 1 LEADER]: Becoming Leader. State: Replica: 70039d6f48d4425eae48f2aa15e0b77d, State: Running, Role: LEADER
I20260925 15:09:10.575688 10919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70039d6f48d4425eae48f2aa15e0b77d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36947 } }
I20260925 15:09:10.576428 10914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:10.578492 10920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70039d6f48d4425eae48f2aa15e0b77d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70039d6f48d4425eae48f2aa15e0b77d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36947 } } }
I20260925 15:09:10.578549 10921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70039d6f48d4425eae48f2aa15e0b77d. Latest consensus state: current_term: 1 leader_uuid: "70039d6f48d4425eae48f2aa15e0b77d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70039d6f48d4425eae48f2aa15e0b77d" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36947 } } }
I20260925 15:09:10.579008 10921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:10.579272 10920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:10.580864 10927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:10.586050 10927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:10.594254 10927 catalog_manager.cc:1347] Generated new cluster ID: 13317736224147da8c17c7770b7774fe
I20260925 15:09:10.594376 10927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:10.606683 10927 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:10.607915 10927 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:10.625234 10927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d: Generated new TSK 0
I20260925 15:09:10.626073 10927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:10.638842 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.1:0
--local_ip_for_outbound_sockets=127.22.236.1
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36947
--builtin_ntp_servers=127.22.236.20:45241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:09:10.957062 10938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:10.957327 10938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:10.957453 10938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:10.964601 10938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:10.964695 10938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:10.964857 10938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.1
I20260925 15:09:10.973737 10938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:45241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.236.1
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.1
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:10.974957 10938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:10.976617 10938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:10.988876 10943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:10.988824 10946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:10.988907 10944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:10.989557 10938 server_base.cc:1034] running on GCE node
I20260925 15:09:10.990108 10938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:10.991043 10938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:10.992224 10938 hybrid_clock.cc:648] HybridClock initialized: now 1790348950992172 us; error 45 us; skew 500 ppm
I20260925 15:09:10.992574 10938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:09:10.994758 10938 webserver.cc:460] Webserver started at http://127.22.236.1:37449/ using document root <none> and password file <none>
I20260925 15:09:10.995420 10938 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:10.995517 10938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:10.995812 10938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:10.998610 10938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/data/instance:
uuid: "75c46a1521c8416f86aeeec4b53bd8cc"
format_stamp: "Formatted at 2026-09-25 15:09:10 on dist-test-slave-nsrc"
server_key: "9297ad102709a188f7883578fe62f7b7"
server_key_iv: "c1d298ae0da4d160d610f34258fe6883"
server_key_version: "encryptionkey@0"
I20260925 15:09:10.999351 10938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance:
uuid: "75c46a1521c8416f86aeeec4b53bd8cc"
format_stamp: "Formatted at 2026-09-25 15:09:10 on dist-test-slave-nsrc"
server_key: "9297ad102709a188f7883578fe62f7b7"
server_key_iv: "c1d298ae0da4d160d610f34258fe6883"
server_key_version: "encryptionkey@0"
I20260925 15:09:11.004303 10938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:09:11.007690 10952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:11.009003 10938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:11.009178 10938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/wal
uuid: "75c46a1521c8416f86aeeec4b53bd8cc"
format_stamp: "Formatted at 2026-09-25 15:09:10 on dist-test-slave-nsrc"
server_key: "9297ad102709a188f7883578fe62f7b7"
server_key_iv: "c1d298ae0da4d160d610f34258fe6883"
server_key_version: "encryptionkey@0"
I20260925 15:09:11.009397 10938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:11.044654 10938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:11.045506 10938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:11.045785 10938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:11.046765 10938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:11.048262 10938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:11.048355 10938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:11.048465 10938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:11.048538 10938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:11.078543 10938 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.1:42231
I20260925 15:09:11.078611 11064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.1:42231 every 8 connection(s)
I20260925 15:09:11.079916 10938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/data/info.pb
I20260925 15:09:11.084656 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 10938
I20260925 15:09:11.084853 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-0/wal/instance
I20260925 15:09:11.085352 23472 external_mini_cluster.cc:1428] Setting key b8bd873a0d238ba2dda21f52d448dd9d
I20260925 15:09:11.093919 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.2:0
--local_ip_for_outbound_sockets=127.22.236.2
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36947
--builtin_ntp_servers=127.22.236.20:45241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:09:11.095402 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:11.082544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.1:57337 (local address 127.22.236.62:36947)
0925 15:09:11.082971 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:11.082976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:11.084321 (+ 1345us) server_negotiation.cc:407] Connection header received
0925 15:09:11.086543 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:11.086549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:11.086631 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:11.086724 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:11.087844 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:11.088826 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:11.091066 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:11.091638 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:11.091754 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:11.094429 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:11.094442 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:11.094447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:11.094490 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:11.094539 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:11.094558 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:11.094633 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:11.094965 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 15:09:11.095071 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":198,"threads_started":1}
I20260925 15:09:11.096774 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:11.083149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36947 (local address 127.22.236.1:57337)
0925 15:09:11.084100 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:11.084176 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:11.086275 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:11.086995 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:11.087006 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:11.087124 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:11.087678 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:11.087696 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:11.089007 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:11.089013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:11.090894 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:11.090910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:11.092055 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:11.092063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:11.092392 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:11.093205 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:09:11.093209 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:11.093289 (+ 80us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:09:11.093458 (+ 169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:11.094696 (+ 1238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:11.094707 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:11.094716 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:09:11.094903 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 15:09:11.095324 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":253,"threads_started":1}
I20260925 15:09:11.098332 11065 heartbeater.cc:347] Connected to a master server at 127.22.236.62:36947
I20260925 15:09:11.098660 11065 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:11.099522 11065 heartbeater.cc:511] Master 127.22.236.62:36947 requested a full tablet report, sending...
I20260925 15:09:11.101739 10879 ts_manager.cc:194] Registered new tserver with Master: 75c46a1521c8416f86aeeec4b53bd8cc (127.22.236.1:42231)
I20260925 15:09:11.103430 10879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.236.1:57337
W20260925 15:09:11.405527 11069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:11.405795 11069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:11.405889 11069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:11.413097 11069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:11.413201 11069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:11.413322 11069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.2
I20260925 15:09:11.422264 11069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:45241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.236.2
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.2
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:11.423480 11069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:11.425169 11069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:11.437147 11074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:11.437219 11075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:11.437188 11077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:11.438282 11069 server_base.cc:1034] running on GCE node
I20260925 15:09:11.438876 11069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:11.439766 11069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:11.440968 11069 hybrid_clock.cc:648] HybridClock initialized: now 1790348951440922 us; error 44 us; skew 500 ppm
I20260925 15:09:11.441258 11069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:09:11.443401 11069 webserver.cc:460] Webserver started at http://127.22.236.2:44049/ using document root <none> and password file <none>
I20260925 15:09:11.444044 11069 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:11.444160 11069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:11.444447 11069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:11.447157 11069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/data/instance:
uuid: "31b2487a2eca43d6891316b006d54ca6"
format_stamp: "Formatted at 2026-09-25 15:09:11 on dist-test-slave-nsrc"
server_key: "a5d0a3dc03b267774905d1a97f8e2985"
server_key_iv: "9a6911654bd21eabd61adf525adb38b4"
server_key_version: "encryptionkey@0"
I20260925 15:09:11.447940 11069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance:
uuid: "31b2487a2eca43d6891316b006d54ca6"
format_stamp: "Formatted at 2026-09-25 15:09:11 on dist-test-slave-nsrc"
server_key: "a5d0a3dc03b267774905d1a97f8e2985"
server_key_iv: "9a6911654bd21eabd61adf525adb38b4"
server_key_version: "encryptionkey@0"
I20260925 15:09:11.453320 11069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 15:09:11.456537 11083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:11.457746 11069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 15:09:11.457904 11069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/wal
uuid: "31b2487a2eca43d6891316b006d54ca6"
format_stamp: "Formatted at 2026-09-25 15:09:11 on dist-test-slave-nsrc"
server_key: "a5d0a3dc03b267774905d1a97f8e2985"
server_key_iv: "9a6911654bd21eabd61adf525adb38b4"
server_key_version: "encryptionkey@0"
I20260925 15:09:11.458128 11069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:11.476039 11069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:11.476861 11069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:11.477150 11069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:11.478067 11069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:11.479478 11069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:11.479573 11069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:11.479677 11069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:11.479746 11069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:11.508821 11069 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.2:43207
I20260925 15:09:11.508895 11195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.2:43207 every 8 connection(s)
I20260925 15:09:11.510238 11069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/data/info.pb
I20260925 15:09:11.519179 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 11069
I20260925 15:09:11.519318 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/wal/instance
I20260925 15:09:11.519862 23472 external_mini_cluster.cc:1428] Setting key 8ffa89f629984d5d632ffb8355a403af
I20260925 15:09:11.522279 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:11.512723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.2:56133 (local address 127.22.236.62:36947)
0925 15:09:11.512887 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:11.512892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:11.514274 (+ 1382us) server_negotiation.cc:407] Connection header received
0925 15:09:11.515973 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:11.515978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:11.516065 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:11.516174 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:11.517202 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:11.517919 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:11.519434 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:11.520060 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:11.520219 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:11.521490 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:11.521495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:11.521498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:11.521516 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:11.521553 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:11.521567 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:11.521650 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:11.521877 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 15:09:11.521991 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:09:11.522924 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.236.3:0
--local_ip_for_outbound_sockets=127.22.236.3
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36947
--builtin_ntp_servers=127.22.236.20:45241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:09:11.523284 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:11.513215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36947 (local address 127.22.236.2:56133)
0925 15:09:11.514130 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:11.514166 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:11.515716 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:11.516408 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:11.516418 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:11.516502 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:11.517043 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:11.517060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:11.518062 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:11.518066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:11.519281 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:11.519296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:11.520302 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:11.520310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:11.520524 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:11.521162 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:09:11.521165 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:11.521216 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:09:11.521317 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:11.521663 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:11.521674 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:11.521682 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:09:11.521867 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 15:09:11.522185 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":281,"threads_started":1}
I20260925 15:09:11.524621 11196 heartbeater.cc:347] Connected to a master server at 127.22.236.62:36947
I20260925 15:09:11.524912 11196 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:11.525610 11196 heartbeater.cc:511] Master 127.22.236.62:36947 requested a full tablet report, sending...
I20260925 15:09:11.526968 10879 ts_manager.cc:194] Registered new tserver with Master: 31b2487a2eca43d6891316b006d54ca6 (127.22.236.2:43207)
I20260925 15:09:11.527760 10879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.236.2:56133
W20260925 15:09:11.834643 11199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:11.834928 11199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:11.835040 11199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:11.842271 11199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:11.842374 11199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:11.842509 11199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.236.3
I20260925 15:09:11.851678 11199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:45241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.236.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.236.3
--webserver_port=0
--tserver_master_addrs=127.22.236.62:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.236.3
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:11.853016 11199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:11.854811 11199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:11.867372 11204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:11.867441 11205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:11.868076 11207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:11.868750 11199 server_base.cc:1034] running on GCE node
I20260925 15:09:11.869318 11199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:11.870208 11199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:11.871429 11199 hybrid_clock.cc:648] HybridClock initialized: now 1790348951871366 us; error 52 us; skew 500 ppm
I20260925 15:09:11.871750 11199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:09:11.874030 11199 webserver.cc:460] Webserver started at http://127.22.236.3:37347/ using document root <none> and password file <none>
I20260925 15:09:11.874692 11199 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:11.874799 11199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:11.875083 11199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:11.877965 11199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/data/instance:
uuid: "23785b69e9304a81a0c453595779b414"
format_stamp: "Formatted at 2026-09-25 15:09:11 on dist-test-slave-nsrc"
server_key: "837c8b883fc42d363213b5987abc208d"
server_key_iv: "38cd6c1175f58b515a0844f091d8e63d"
server_key_version: "encryptionkey@0"
I20260925 15:09:11.878701 11199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance:
uuid: "23785b69e9304a81a0c453595779b414"
format_stamp: "Formatted at 2026-09-25 15:09:11 on dist-test-slave-nsrc"
server_key: "837c8b883fc42d363213b5987abc208d"
server_key_iv: "38cd6c1175f58b515a0844f091d8e63d"
server_key_version: "encryptionkey@0"
I20260925 15:09:11.883618 11199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 15:09:11.887017 11213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:11.888362 11199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:11.888505 11199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/wal
uuid: "23785b69e9304a81a0c453595779b414"
format_stamp: "Formatted at 2026-09-25 15:09:11 on dist-test-slave-nsrc"
server_key: "837c8b883fc42d363213b5987abc208d"
server_key_iv: "38cd6c1175f58b515a0844f091d8e63d"
server_key_version: "encryptionkey@0"
I20260925 15:09:11.888691 11199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:11.909008 11199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:11.909938 11199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:11.910224 11199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:11.911332 11199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:11.912947 11199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:11.913043 11199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:11.913149 11199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:11.913214 11199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:11.941901 11199 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.3:45645
I20260925 15:09:11.941978 11325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.3:45645 every 8 connection(s)
I20260925 15:09:11.943122 11199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/data/info.pb
I20260925 15:09:11.945510 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 11199
I20260925 15:09:11.945698 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/wal/instance
I20260925 15:09:11.946105 23472 external_mini_cluster.cc:1428] Setting key a956a1a215ee071c18399fb250960aa7
I20260925 15:09:11.955010 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:11.945582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:59731 (local address 127.22.236.62:36947)
0925 15:09:11.945788 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:11.945793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:11.947228 (+ 1435us) server_negotiation.cc:407] Connection header received
0925 15:09:11.949098 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:11.949102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:11.949167 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:11.949263 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:11.950318 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:11.951034 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:11.952625 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:11.953013 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:11.953111 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:11.954349 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:11.954355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:11.954358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:11.954396 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:11.954428 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:11.954440 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:11.954514 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:11.954677 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 15:09:11.954784 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:09:11.955960 11328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:11.946068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36947 (local address 127.22.236.3:59731)
0925 15:09:11.947081 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:11.947131 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:11.948858 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:11.949503 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:11.949511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:11.949598 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:11.950158 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:11.950175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:11.951184 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:11.951188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:11.952505 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:11.952518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:11.953177 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:11.953182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:11.953384 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:11.954030 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:09:11.954032 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:11.954072 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:09:11.954154 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:11.954519 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:11.954525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:11.954531 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:09:11.954664 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 15:09:11.954967 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":309,"threads_started":1}
I20260925 15:09:11.957219 11326 heartbeater.cc:347] Connected to a master server at 127.22.236.62:36947
I20260925 15:09:11.957495 11326 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:11.958218 11326 heartbeater.cc:511] Master 127.22.236.62:36947 requested a full tablet report, sending...
I20260925 15:09:11.959563 10879 ts_manager.cc:194] Registered new tserver with Master: 23785b69e9304a81a0c453595779b414 (127.22.236.3:45645)
I20260925 15:09:11.960495 10879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.236.3:59731
I20260925 15:09:11.961176 23472 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:09:11.970074 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:11.965164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34466 (local address 127.22.236.62:36947)
0925 15:09:11.965304 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:11.965308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:11.965658 (+ 350us) server_negotiation.cc:407] Connection header received
0925 15:09:11.965781 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:11.965785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:11.965850 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:11.965948 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:11.966322 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:11.966995 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:11.968248 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:11.968644 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:11.968730 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:11.969587 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:11.969590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:11.969593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:11.969608 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:11.969631 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:11.969641 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:11.969696 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:11.969777 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 15:09:11.969843 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:09:11.973762 10878 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:09:11.980548 10878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:09:12.026305 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:11.991699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:43207 (local address 127.0.0.1:53300)
0925 15:09:12.017864 (+ 26165us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:12.017883 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:12.018002 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:12.018969 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:12.018976 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:12.019005 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:12.019477 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:12.019499 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.021561 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.021566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:12.023462 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:12.023474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.025324 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.025330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:12.025529 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:12.025585 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 15:09:12.025709 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 15:09:12.025916 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26011,"mutex_wait_us":138,"thread_start_us":24887,"threads_started":1}
I20260925 15:09:12.026494 11343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:11.992278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.22.236.1:42231)
0925 15:09:12.016077 (+ 23799us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:12.016088 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:12.018129 (+ 2041us) server_negotiation.cc:407] Connection header received
0925 15:09:12.018421 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:12.018430 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:12.018658 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:12.018812 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:12.019827 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.021390 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.024242 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.025154 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.025267 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:12.025523 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:12.025996 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 15:09:12.026167 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23601,"thread_start_us":23808,"threads_started":1}
I20260925 15:09:12.026576 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:11.991469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.3:45645 (local address 127.0.0.1:54244)
0925 15:09:12.016863 (+ 25394us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:12.016912 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:12.017086 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:12.019313 (+ 2227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:12.019319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:12.019339 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:12.019790 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:12.019800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.021309 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.021312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:12.023975 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:12.023984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.025765 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.025772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:12.025965 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:12.025997 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:09:12.026057 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:09:12.026185 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25132,"mutex_wait_us":172,"thread_start_us":26307,"threads_started":1}
I20260925 15:09:12.027242 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:11.992300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.22.236.2:43207)
0925 15:09:12.017636 (+ 25336us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:12.017644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:12.018015 (+ 371us) server_negotiation.cc:407] Connection header received
0925 15:09:12.018150 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:12.018159 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:12.018446 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:12.018637 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:12.019756 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.021382 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.023692 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.025133 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.025267 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:12.025522 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:12.026774 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0925 15:09:12.026941 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25155,"thread_start_us":25789,"threads_started":1}
I20260925 15:09:12.027462 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:11.991659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.22.236.3:45645)
0925 15:09:12.018787 (+ 27128us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:12.018795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:12.018825 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:09:12.018911 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:12.018917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:12.019113 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:12.019896 (+ 783us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:12.020104 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.021182 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.024208 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.025554 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.026675 (+ 1121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:12.026916 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:12.026982 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 15:09:12.027129 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26967,"thread_start_us":27897,"threads_started":1}
I20260925 15:09:12.027165 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:11.991990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:42231 (local address 127.0.0.1:42898)
0925 15:09:12.018029 (+ 26039us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:12.018043 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:12.018220 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:12.019113 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:12.019118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:12.019147 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:12.019662 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:12.019675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.021513 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.021518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:12.023967 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:12.023982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.025645 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.025651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:12.025808 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:12.025838 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:09:12.026724 (+ 886us) client_negotiation.cc:240] Negotiation successful
0925 15:09:12.026845 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25933,"mutex_wait_us":157,"thread_start_us":25576,"threads_started":1}
I20260925 15:09:12.030484 11131 tablet_service.cc:1467] Processing CreateTablet for tablet 6be3b881263b41ca8ed5e3cca2de9dc2 (DEFAULT_TABLE table=test-table [id=e8fd09d2c141433490518c9138920a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:12.031044 11261 tablet_service.cc:1467] Processing CreateTablet for tablet 6be3b881263b41ca8ed5e3cca2de9dc2 (DEFAULT_TABLE table=test-table [id=e8fd09d2c141433490518c9138920a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:12.031044 11000 tablet_service.cc:1467] Processing CreateTablet for tablet 6be3b881263b41ca8ed5e3cca2de9dc2 (DEFAULT_TABLE table=test-table [id=e8fd09d2c141433490518c9138920a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:12.031589 11131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be3b881263b41ca8ed5e3cca2de9dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:12.032351 11261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be3b881263b41ca8ed5e3cca2de9dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:12.032351 11000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be3b881263b41ca8ed5e3cca2de9dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:12.042352 11344 tablet_bootstrap.cc:492] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6: Bootstrap starting.
I20260925 15:09:12.046041 11345 tablet_bootstrap.cc:492] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414: Bootstrap starting.
I20260925 15:09:12.046401 11344 tablet_bootstrap.cc:654] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:12.047411 11346 tablet_bootstrap.cc:492] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc: Bootstrap starting.
I20260925 15:09:12.048117 11344 log.cc:824] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:12.049358 11345 tablet_bootstrap.cc:654] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:12.050329 11346 tablet_bootstrap.cc:654] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:12.050525 11345 log.cc:824] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:12.051256 11344 tablet_bootstrap.cc:492] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6: No bootstrap required, opened a new log
I20260925 15:09:12.051297 11346 log.cc:824] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:12.051518 11344 ts_tablet_manager.cc:1397] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260925 15:09:12.053472 11345 tablet_bootstrap.cc:492] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414: No bootstrap required, opened a new log
I20260925 15:09:12.053727 11345 ts_tablet_manager.cc:1397] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:09:12.055406 11346 tablet_bootstrap.cc:492] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc: No bootstrap required, opened a new log
I20260925 15:09:12.055634 11346 ts_tablet_manager.cc:1397] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:09:12.062286 11344 raft_consensus.cc:348] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23785b69e9304a81a0c453595779b414" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45645 } } peers { permanent_uuid: "31b2487a2eca43d6891316b006d54ca6" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 43207 } } peers { permanent_uuid: "75c46a1521c8416f86aeeec4b53bd8cc" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 42231 } }
I20260925 15:09:12.062772 11344 raft_consensus.cc:374] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:12.062918 11344 raft_consensus.cc:729] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b2487a2eca43d6891316b006d54ca6, State: Initialized, Role: FOLLOWER
I20260925 15:09:12.062922 11346 raft_consensus.cc:348] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23785b69e9304a81a0c453595779b414" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45645 } } peers { permanent_uuid: "31b2487a2eca43d6891316b006d54ca6" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 43207 } } peers { permanent_uuid: "75c46a1521c8416f86aeeec4b53bd8cc" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 42231 } }
I20260925 15:09:12.063371 11346 raft_consensus.cc:374] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:12.063472 11346 raft_consensus.cc:729] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c46a1521c8416f86aeeec4b53bd8cc, State: Initialized, Role: FOLLOWER
I20260925 15:09:12.063479 11345 raft_consensus.cc:348] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23785b69e9304a81a0c453595779b414" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45645 } } peers { permanent_uuid: "31b2487a2eca43d6891316b006d54ca6" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 43207 } } peers { permanent_uuid: "75c46a1521c8416f86aeeec4b53bd8cc" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 42231 } }
I20260925 15:09:12.063560 11344 consensus_queue.cc:260] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23785b69e9304a81a0c453595779b414" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45645 } } peers { permanent_uuid: "31b2487a2eca43d6891316b006d54ca6" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 43207 } } peers { permanent_uuid: "75c46a1521c8416f86aeeec4b53bd8cc" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 42231 } }
I20260925 15:09:12.063925 11345 raft_consensus.cc:374] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:12.064026 11345 raft_consensus.cc:729] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23785b69e9304a81a0c453595779b414, State: Initialized, Role: FOLLOWER
I20260925 15:09:12.064074 11346 consensus_queue.cc:260] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23785b69e9304a81a0c453595779b414" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45645 } } peers { permanent_uuid: "31b2487a2eca43d6891316b006d54ca6" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 43207 } } peers { permanent_uuid: "75c46a1521c8416f86aeeec4b53bd8cc" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 42231 } }
I20260925 15:09:12.064759 11345 consensus_queue.cc:260] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23785b69e9304a81a0c453595779b414" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45645 } } peers { permanent_uuid: "31b2487a2eca43d6891316b006d54ca6" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 43207 } } peers { permanent_uuid: "75c46a1521c8416f86aeeec4b53bd8cc" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 42231 } }
I20260925 15:09:12.065074 11065 heartbeater.cc:503] Master 127.22.236.62:36947 was elected leader, sending a full tablet report...
I20260925 15:09:12.066294 11326 heartbeater.cc:503] Master 127.22.236.62:36947 was elected leader, sending a full tablet report...
I20260925 15:09:12.066516 11345 ts_tablet_manager.cc:1428] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260925 15:09:12.067842 11346 ts_tablet_manager.cc:1428] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260925 15:09:12.069284 11196 heartbeater.cc:503] Master 127.22.236.62:36947 was elected leader, sending a full tablet report...
I20260925 15:09:12.069890 11344 ts_tablet_manager.cc:1428] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6: Time spent starting tablet: real 0.018s user 0.011s sys 0.006s
I20260925 15:09:12.081823 11352 raft_consensus.cc:482] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 15:09:12.082116 11066 tablet.cc:2369] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:09:12.082103 11352 raft_consensus.cc:504] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23785b69e9304a81a0c453595779b414" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45645 } } peers { permanent_uuid: "31b2487a2eca43d6891316b006d54ca6" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 43207 } } peers { permanent_uuid: "75c46a1521c8416f86aeeec4b53bd8cc" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 42231 } }
I20260925 15:09:12.084451 11352 leader_election.cc:288] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b2487a2eca43d6891316b006d54ca6 (127.22.236.2:43207), 75c46a1521c8416f86aeeec4b53bd8cc (127.22.236.1:42231)
I20260925 15:09:12.089792 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.084091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:56273 (local address 127.22.236.2:43207)
0925 15:09:12.084314 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:12.084319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:12.084341 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:09:12.084579 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:12.084584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:12.084657 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:12.084751 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:12.085245 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.086218 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.087734 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.088921 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.089055 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:12.089121 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:12.089439 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 15:09:12.089556 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260925 15:09:12.089825 11328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.084016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.2:43207 (local address 127.22.236.3:56273)
0925 15:09:12.084280 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:12.084299 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:12.084409 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:12.084802 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:12.084807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:12.084832 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:12.085039 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:12.085054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.086305 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.086310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:12.087581 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:12.087594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.089090 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.089095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:12.089276 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:12.089309 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:09:12.089408 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 15:09:12.089541 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260925 15:09:12.090940 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.084689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:42231 (local address 127.22.236.3:55619)
0925 15:09:12.085363 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:12.085378 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:12.085486 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:12.085781 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:12.085786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:12.085808 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:12.086129 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:12.086140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.087502 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.087508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:12.088889 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:12.088901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.090368 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.090381 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:12.090539 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:12.090570 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:09:12.090661 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:09:12.090745 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":370,"threads_started":1}
I20260925 15:09:12.091123 11343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.084466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.236.3:55619 (local address 127.22.236.1:42231)
0925 15:09:12.084630 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:12.084636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:12.085455 (+ 819us) server_negotiation.cc:407] Connection header received
0925 15:09:12.085589 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:12.085594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:12.085678 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:12.085760 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:12.086272 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.087333 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.089051 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.090225 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.090367 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:12.090430 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:12.090750 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 15:09:12.090895 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:09:12.091042 11151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be3b881263b41ca8ed5e3cca2de9dc2" candidate_uuid: "23785b69e9304a81a0c453595779b414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b2487a2eca43d6891316b006d54ca6" is_pre_election: true
I20260925 15:09:12.091531 11151 raft_consensus.cc:2454] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23785b69e9304a81a0c453595779b414 in term 0.
I20260925 15:09:12.091956 11020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be3b881263b41ca8ed5e3cca2de9dc2" candidate_uuid: "23785b69e9304a81a0c453595779b414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c46a1521c8416f86aeeec4b53bd8cc" is_pre_election: true
I20260925 15:09:12.092243 11215 leader_election.cc:302] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23785b69e9304a81a0c453595779b414, 31b2487a2eca43d6891316b006d54ca6; no voters:
I20260925 15:09:12.092458 11020 raft_consensus.cc:2454] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23785b69e9304a81a0c453595779b414 in term 0.
I20260925 15:09:12.092605 11352 raft_consensus.cc:2781] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:09:12.092728 11352 raft_consensus.cc:482] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:09:12.092846 11352 raft_consensus.cc:3037] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:12.095330 11352 raft_consensus.cc:504] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23785b69e9304a81a0c453595779b414" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45645 } } peers { permanent_uuid: "31b2487a2eca43d6891316b006d54ca6" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 43207 } } peers { permanent_uuid: "75c46a1521c8416f86aeeec4b53bd8cc" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 42231 } }
I20260925 15:09:12.095897 11352 leader_election.cc:288] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [CANDIDATE]: Term 1 election: Requested vote from peers 31b2487a2eca43d6891316b006d54ca6 (127.22.236.2:43207), 75c46a1521c8416f86aeeec4b53bd8cc (127.22.236.1:42231)
I20260925 15:09:12.096266 11151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be3b881263b41ca8ed5e3cca2de9dc2" candidate_uuid: "23785b69e9304a81a0c453595779b414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b2487a2eca43d6891316b006d54ca6"
I20260925 15:09:12.096310 11020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be3b881263b41ca8ed5e3cca2de9dc2" candidate_uuid: "23785b69e9304a81a0c453595779b414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c46a1521c8416f86aeeec4b53bd8cc"
I20260925 15:09:12.096446 11151 raft_consensus.cc:3037] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:12.096504 11020 raft_consensus.cc:3037] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:12.098572 11151 raft_consensus.cc:2454] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23785b69e9304a81a0c453595779b414 in term 1.
I20260925 15:09:12.098600 11020 raft_consensus.cc:2454] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23785b69e9304a81a0c453595779b414 in term 1.
I20260925 15:09:12.098905 11215 leader_election.cc:302] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23785b69e9304a81a0c453595779b414, 31b2487a2eca43d6891316b006d54ca6; no voters:
I20260925 15:09:12.099190 11352 raft_consensus.cc:2781] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:12.099596 11352 raft_consensus.cc:686] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [term 1 LEADER]: Becoming Leader. State: Replica: 23785b69e9304a81a0c453595779b414, State: Running, Role: LEADER
I20260925 15:09:12.100097 11352 consensus_queue.cc:237] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23785b69e9304a81a0c453595779b414" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45645 } } peers { permanent_uuid: "31b2487a2eca43d6891316b006d54ca6" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 43207 } } peers { permanent_uuid: "75c46a1521c8416f86aeeec4b53bd8cc" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 42231 } }
I20260925 15:09:12.104249 10877 catalog_manager.cc:5495] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23785b69e9304a81a0c453595779b414 (127.22.236.3). New cstate: current_term: 1 leader_uuid: "23785b69e9304a81a0c453595779b414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23785b69e9304a81a0c453595779b414" member_type: VOTER last_known_addr { host: "127.22.236.3" port: 45645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b2487a2eca43d6891316b006d54ca6" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 43207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75c46a1521c8416f86aeeec4b53bd8cc" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 42231 } health_report { overall_health: UNKNOWN } } }
I20260925 15:09:12.128551 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.123541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.22.236.3:45645)
0925 15:09:12.123726 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:12.123732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:12.123753 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:09:12.123973 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:12.123977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:12.124052 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:12.124198 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:12.124550 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.125254 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.126570 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.126979 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.127080 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:12.127515 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:12.127995 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:12.128130 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:12.128246 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 15:09:12.128320 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:09:12.136204 11151 raft_consensus.cc:1261] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6 [term 1 FOLLOWER]: Refusing update from remote peer 23785b69e9304a81a0c453595779b414: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:12.136202 11020 raft_consensus.cc:1261] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc [term 1 FOLLOWER]: Refusing update from remote peer 23785b69e9304a81a0c453595779b414: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:12.137212 11354 consensus_queue.cc:1035] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c46a1521c8416f86aeeec4b53bd8cc" member_type: VOTER last_known_addr { host: "127.22.236.1" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:12.137562 11352 consensus_queue.cc:1035] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b2487a2eca43d6891316b006d54ca6" member_type: VOTER last_known_addr { host: "127.22.236.2" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:12.147271 11357 mvcc.cc:205] Tried to move back new op lower bound from 7333269307939725312 to 7333269307805896704. Current Snapshot: MvccSnapshot[applied={T|T < 7333269307939725312}]
I20260925 15:09:12.149317 11359 mvcc.cc:205] Tried to move back new op lower bound from 7333269307939725312 to 7333269307805896704. Current Snapshot: MvccSnapshot[applied={T|T < 7333269307939725312}]
I20260925 15:09:12.152328 11358 mvcc.cc:205] Tried to move back new op lower bound from 7333269307939725312 to 7333269307805896704. Current Snapshot: MvccSnapshot[applied={T|T < 7333269307939725312}]
I20260925 15:09:12.162925 10878 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:09:12.163241 10878 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:09:12.166601 10878 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70039d6f48d4425eae48f2aa15e0b77d: Sending DeleteTablet for 3 replicas of tablet 6be3b881263b41ca8ed5e3cca2de9dc2
I20260925 15:09:12.167584 11261 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be3b881263b41ca8ed5e3cca2de9dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:12 UTC) from {username='slave'} at 127.0.0.1:54244
I20260925 15:09:12.167670 11131 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be3b881263b41ca8ed5e3cca2de9dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:12 UTC) from {username='slave'} at 127.0.0.1:53300
I20260925 15:09:12.168023 11000 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be3b881263b41ca8ed5e3cca2de9dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:12 UTC) from {username='slave'} at 127.0.0.1:42898
I20260925 15:09:12.169205 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 10938
I20260925 15:09:12.178609 11366 tablet_replica.cc:330] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414: stopping tablet replica
I20260925 15:09:12.179059 11366 raft_consensus.cc:2229] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:09:12.179764 11366 raft_consensus.cc:2258] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:12.180540 11367 tablet_replica.cc:330] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6: stopping tablet replica
I20260925 15:09:12.180927 11367 raft_consensus.cc:2229] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:09:12.181445 11367 raft_consensus.cc:2258] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:12.183633 11367 ts_tablet_manager.cc:1905] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:12.186916 11368 tablet_replica.cc:330] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc: stopping tablet replica
I20260925 15:09:12.187310 11368 raft_consensus.cc:2229] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:09:12.187788 11368 raft_consensus.cc:2258] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:12.189159 11343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.170541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.22.236.1:42231)
0925 15:09:12.170748 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:12.170756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:12.176923 (+ 6167us) server_negotiation.cc:407] Connection header received
0925 15:09:12.177205 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:12.177211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:12.177338 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:12.177428 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:12.178142 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.179262 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.181112 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.181844 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.181985 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:12.187481 (+ 5496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:12.187491 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:12.187508 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:12.188297 (+ 789us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:12.188354 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:12.188388 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:12.188493 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:12.188694 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 15:09:12.188813 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:09:12.190204 11060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:12.191895 11367 ts_tablet_manager.cc:1918] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:09:12.192055 11367 log.cc:1196] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-1/wal/wals/6be3b881263b41ca8ed5e3cca2de9dc2
I20260925 15:09:12.192117 11366 ts_tablet_manager.cc:1905] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:12.192629 11367 ts_tablet_manager.cc:1939] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 31b2487a2eca43d6891316b006d54ca6: Deleting consensus metadata
I20260925 15:09:12.194630 10864 catalog_manager.cc:4841] TS 31b2487a2eca43d6891316b006d54ca6 (127.22.236.2:43207): tablet 6be3b881263b41ca8ed5e3cca2de9dc2 (table test-table [id=e8fd09d2c141433490518c9138920a70]) successfully deleted
I20260925 15:09:12.201221 11366 ts_tablet_manager.cc:1918] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:09:12.201403 11366 log.cc:1196] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414: Deleting WAL directory at /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348645297121-23472-0/minicluster-data/ts-2/wal/wals/6be3b881263b41ca8ed5e3cca2de9dc2
I20260925 15:09:12.201967 11366 ts_tablet_manager.cc:1939] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 23785b69e9304a81a0c453595779b414: Deleting consensus metadata
I20260925 15:09:12.203801 10865 catalog_manager.cc:4841] TS 23785b69e9304a81a0c453595779b414 (127.22.236.3:45645): tablet 6be3b881263b41ca8ed5e3cca2de9dc2 (table test-table [id=e8fd09d2c141433490518c9138920a70]) successfully deleted
I20260925 15:09:12.191548 11368 ts_tablet_manager.cc:1905] T 6be3b881263b41ca8ed5e3cca2de9dc2 P 75c46a1521c8416f86aeeec4b53bd8cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:12.281163 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 10938
I20260925 15:09:12.295779 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 11069
W20260925 15:09:12.295822 10863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.236.1:42231 (error 108)
I20260925 15:09:12.297746 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.297294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:42231 (local address 127.0.0.1:42916)
0925 15:09:12.297458 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:12.297578 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:09:12.298833 10863 catalog_manager.cc:4584] TS 75c46a1521c8416f86aeeec4b53bd8cc (127.22.236.1:42231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6be3b881263b41ca8ed5e3cca2de9dc2: Network error: Client connection negotiation failed: client connection to 127.22.236.1:42231: connect: Connection refused (error 111)
I20260925 15:09:12.303228 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.296932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.22.236.2:43207)
0925 15:09:12.297107 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:12.297113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:12.297133 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:09:12.297196 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:12.297202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:12.297327 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:12.297432 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:12.297936 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.298638 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.300446 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.300828 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.300943 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:12.302005 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:12.302013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:12.302026 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:12.302561 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:12.302606 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:12.302625 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:12.302705 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:12.302835 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 15:09:12.302935 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:09:12.303844 11191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:12.352797 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.352261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:42231 (local address 127.0.0.1:42922)
0925 15:09:12.352498 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:12.352606 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 15:09:12.353922 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.353534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:42231 (local address 127.0.0.1:42932)
0925 15:09:12.353683 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:12.353769 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:09:12.391785 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 11069
I20260925 15:09:12.406114 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 11199
I20260925 15:09:12.414394 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.413851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:42231 (local address 127.0.0.1:42942)
0925 15:09:12.414088 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:12.414194 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 15:09:12.415416 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.415021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:42231 (local address 127.0.0.1:42948)
0925 15:09:12.415170 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:12.415269 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:09:12.419188 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.407108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54258 (local address 127.22.236.3:45645)
0925 15:09:12.407263 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:12.407267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:12.407284 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:09:12.407337 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:12.407341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:12.407425 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:12.407506 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:12.408011 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.408825 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.410690 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:12.411187 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:12.411281 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:12.417806 (+ 6525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:12.417815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:12.417829 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:12.418436 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:12.418484 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:12.418514 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:12.418597 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:12.418777 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 15:09:12.418866 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:09:12.419914 11321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:12.495852 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.495366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:42231 (local address 127.0.0.1:42960)
0925 15:09:12.495582 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:12.495674 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 15:09:12.496914 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.496512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:42231 (local address 127.0.0.1:42970)
0925 15:09:12.496696 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:12.496805 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 15:09:12.541277 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 11199
I20260925 15:09:12.564050 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 10848
I20260925 15:09:12.565119 10909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:12.672761 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:12.672112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.1:42231 (local address 127.0.0.1:42972)
0925 15:09:12.672418 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:12.672560 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.1:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260925 15:09:12.672786 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 10848
2026-09-25T15:09:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2602 ms)
[----------] 6 tests from EncryptionPolicyTest (20888 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:12 dist-test-slave-nsrc krb5kdc[11377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:12 dist-test-slave-nsrc krb5kdc[11377](info): set up 2 sockets
Sep 25 15:09:12 dist-test-slave-nsrc krb5kdc[11377](info): commencing operation
krb5kdc: starting...
W20260925 15:09:13.696514 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:13 dist-test-slave-nsrc krb5kdc[11377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:09:13.919319 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:14.230728 11390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:14.230990 11390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:14.231082 11390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:14.237988 11390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:14.238089 11390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:14.238142 11390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:14.238200 11390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:14.238241 11390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:14.247745 11390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33713
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:14.248975 11390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:14.250186 11390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:14.261147 11395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:14.261176 11396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:14.261225 11398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:14.261804 11390 server_base.cc:1034] running on GCE node
I20260925 15:09:14.262492 11390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:14.263760 11390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:14.264941 11390 hybrid_clock.cc:648] HybridClock initialized: now 1790348954264908 us; error 25 us; skew 500 ppm
I20260925 15:09:14.265240 11390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:14 dist-test-slave-nsrc krb5kdc[11377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:14.268044 11390 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:09:14.269521 11390 webserver.cc:460] Webserver started at http://127.0.0.1:35727/ using document root <none> and password file <none>
I20260925 15:09:14.270159 11390 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:14.270277 11390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:14.270581 11390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:14.273458 11390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "95aeab34e65d423c8bc033010aacc601"
format_stamp: "Formatted at 2026-09-25 15:09:14 on dist-test-slave-nsrc"
server_key: "5ada859f5044b1fb2792687417570f6f"
server_key_iv: "f206f0d6e1e4196b5be96f991814e8cb"
server_key_version: "encryptionkey@0"
I20260925 15:09:14.274159 11390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "95aeab34e65d423c8bc033010aacc601"
format_stamp: "Formatted at 2026-09-25 15:09:14 on dist-test-slave-nsrc"
server_key: "5ada859f5044b1fb2792687417570f6f"
server_key_iv: "f206f0d6e1e4196b5be96f991814e8cb"
server_key_version: "encryptionkey@0"
I20260925 15:09:14.279006 11390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:09:14.282258 11405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:14.283464 11390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:09:14.283615 11390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "95aeab34e65d423c8bc033010aacc601"
format_stamp: "Formatted at 2026-09-25 15:09:14 on dist-test-slave-nsrc"
server_key: "5ada859f5044b1fb2792687417570f6f"
server_key_iv: "f206f0d6e1e4196b5be96f991814e8cb"
server_key_version: "encryptionkey@0"
I20260925 15:09:14.283778 11390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:14.293942 11390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:14.297899 11390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:14.298173 11390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:14.314781 11390 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33713
I20260925 15:09:14.314813 11456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33713 every 8 connection(s)
I20260925 15:09:14.316051 11390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:14.319367 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 11390
I20260925 15:09:14.319543 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:14.319612 11457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:14.320258 23472 external_mini_cluster.cc:1428] Setting key 70f0afb57a6e9bd10db8425e3d7d2545
I20260925 15:09:14.333178 11457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601: Bootstrap starting.
Sep 25 15:09:14 dist-test-slave-nsrc krb5kdc[11377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:09:14.336524 11457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:14.337499 11457 log.cc:824] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:14.340530 11457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601: No bootstrap required, opened a new log
I20260925 15:09:14.342163 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:14.321959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37503 (local address 127.0.0.1:33713)
0925 15:09:14.322714 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:14.322724 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:14.322763 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:09:14.324267 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:14.324290 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:14.324679 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:14.325012 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:14.326640 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:14.327953 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:14.329684 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:14.330362 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:14.330484 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:14.335261 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:14.335268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:14.335282 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:14.335315 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:14.338322 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:14.339162 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:14.339168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:14.339173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:14.339323 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:14.339762 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:14.339765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:14.339767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:14.340265 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:14.340501 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:14.340943 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 15:09:14.341211 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":350,"threads_started":1}
I20260925 15:09:14.345826 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:14.349427 11457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95aeab34e65d423c8bc033010aacc601" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33713 } }
I20260925 15:09:14.349884 11457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:14.349995 11457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95aeab34e65d423c8bc033010aacc601, State: Initialized, Role: FOLLOWER
I20260925 15:09:14.350623 11457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95aeab34e65d423c8bc033010aacc601" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33713 } }
I20260925 15:09:14.350898 11457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:14.351040 11457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:14.351210 11457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:14.354209 11457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95aeab34e65d423c8bc033010aacc601" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33713 } }
I20260925 15:09:14.354701 11457 leader_election.cc:302] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95aeab34e65d423c8bc033010aacc601; no voters:
I20260925 15:09:14.355307 11457 leader_election.cc:288] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:14.355399 11467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:14.357230 11457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:14.361189 11467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [term 1 LEADER]: Becoming Leader. State: Replica: 95aeab34e65d423c8bc033010aacc601, State: Running, Role: LEADER
I20260925 15:09:14.361799 11467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95aeab34e65d423c8bc033010aacc601" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33713 } }
I20260925 15:09:14.364606 11468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95aeab34e65d423c8bc033010aacc601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95aeab34e65d423c8bc033010aacc601" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33713 } } }
I20260925 15:09:14.365018 11468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:14.365473 11469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95aeab34e65d423c8bc033010aacc601. Latest consensus state: current_term: 1 leader_uuid: "95aeab34e65d423c8bc033010aacc601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95aeab34e65d423c8bc033010aacc601" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33713 } } }
I20260925 15:09:14.365732 11469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:14.368281 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:14.351219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59693 (local address 127.0.0.1:33713)
0925 15:09:14.351403 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:14.351410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:14.351860 (+ 450us) server_negotiation.cc:407] Connection header received
0925 15:09:14.352013 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:14.352016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:14.352078 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:14.352198 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:14.353776 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:14.354522 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:14.356281 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:14.357024 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:14.357156 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:14.360232 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:14.360239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:14.360244 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:14.360284 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:14.364097 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:14.365138 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:14.365143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:14.365146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:14.365287 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:14.365863 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:14.365867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:14.365870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:14.366247 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:14.367508 (+ 1261us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:14.367679 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 15:09:14.367854 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:09:14.373339 11478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:14.377506 11478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:14.385380 11478 catalog_manager.cc:1347] Generated new cluster ID: a9fa0142686d496698b92b601ae2618b
I20260925 15:09:14.385497 11478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:14.397045 11478 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:14.398375 11478 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:14.404582 11478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95aeab34e65d423c8bc033010aacc601: Generated new TSK 0
I20260925 15:09:14.405398 11478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:14.415436 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 11390
I20260925 15:09:14.417459 11452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:14.509925 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 11390
W20260925 15:09:14.523469 11375 connection.cc:535] client connection to 127.0.0.1:33713 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33713): Connection reset by peer (error 104)
2026-09-25T15:09:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1829 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:14 dist-test-slave-nsrc krb5kdc[11490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:14 dist-test-slave-nsrc krb5kdc[11490](info): set up 2 sockets
Sep 25 15:09:14 dist-test-slave-nsrc krb5kdc[11490](info): commencing operation
krb5kdc: starting...
W20260925 15:09:15.536387 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:15 dist-test-slave-nsrc krb5kdc[11490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:09:15.762885 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:16.072068 11503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:16.072372 11503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:16.072512 11503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:16.079499 11503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:16.079615 11503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:16.079694 11503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:16.079770 11503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:16.079831 11503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:16.089569 11503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41505
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:16.090832 11503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:16.092056 11503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:16.102975 11511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:16.103121 11508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:16.103199 11509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:16.103755 11503 server_base.cc:1034] running on GCE node
I20260925 15:09:16.104534 11503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:16.105867 11503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:16.107065 11503 hybrid_clock.cc:648] HybridClock initialized: now 1790348956107021 us; error 38 us; skew 500 ppm
I20260925 15:09:16.107374 11503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:16 dist-test-slave-nsrc krb5kdc[11490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:16.110247 11503 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:09:16.111617 11503 webserver.cc:460] Webserver started at http://127.0.0.1:36723/ using document root <none> and password file <none>
I20260925 15:09:16.112296 11503 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:16.112417 11503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:16.112694 11503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:16.115548 11503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "430315c358644870a30c3358c8e2a5a6"
format_stamp: "Formatted at 2026-09-25 15:09:16 on dist-test-slave-nsrc"
server_key: "b7fffabd5a476abdfbf682dc8bd73698"
server_key_iv: "03d8659f66585e3f78c9d56712af60a7"
server_key_version: "encryptionkey@0"
I20260925 15:09:16.116317 11503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "430315c358644870a30c3358c8e2a5a6"
format_stamp: "Formatted at 2026-09-25 15:09:16 on dist-test-slave-nsrc"
server_key: "b7fffabd5a476abdfbf682dc8bd73698"
server_key_iv: "03d8659f66585e3f78c9d56712af60a7"
server_key_version: "encryptionkey@0"
I20260925 15:09:16.121131 11503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:09:16.124571 11518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:16.125823 11503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 15:09:16.125976 11503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "430315c358644870a30c3358c8e2a5a6"
format_stamp: "Formatted at 2026-09-25 15:09:16 on dist-test-slave-nsrc"
server_key: "b7fffabd5a476abdfbf682dc8bd73698"
server_key_iv: "03d8659f66585e3f78c9d56712af60a7"
server_key_version: "encryptionkey@0"
I20260925 15:09:16.126148 11503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:16.148406 11503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:16.152282 11503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:16.152539 11503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:16.169150 11503 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41505
I20260925 15:09:16.169181 11569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41505 every 8 connection(s)
I20260925 15:09:16.170440 11503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:16.172996 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 11503
I20260925 15:09:16.173170 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:16.173671 23472 external_mini_cluster.cc:1428] Setting key 9dd5d097706d4097d1dca8f6a1fd1cb2
I20260925 15:09:16.174399 11570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:16.187224 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:16.175451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46985 (local address 127.0.0.1:41505)
0925 15:09:16.176321 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:16.176333 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:16.176375 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:09:16.178724 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:16.178745 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:16.179132 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:16.179451 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:16.180029 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:16.181382 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:16.183158 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:16.183786 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:16.183902 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:16.185144 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:16.185151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:16.185164 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:16.185701 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:16.185779 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:16.185802 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:16.185926 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:16.186149 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 15:09:16.186310 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":353,"threads_started":1}
I20260925 15:09:16.187333 11570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6: Bootstrap starting.
I20260925 15:09:16.190351 11570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:16.190939 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:16.191454 11570 log.cc:824] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:16.194579 11570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6: No bootstrap required, opened a new log
I20260925 15:09:16.201767 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:16.195587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59519 (local address 127.0.0.1:41505)
0925 15:09:16.195758 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:16.195763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:16.196106 (+ 343us) server_negotiation.cc:407] Connection header received
0925 15:09:16.196275 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:16.196280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:16.196356 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:16.196462 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:16.196889 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:16.198032 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:16.199236 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:16.199918 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:16.200045 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:16.201046 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:16.201052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:16.201058 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:16.201080 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:16.201124 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:16.201142 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:16.201243 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:16.201365 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 15:09:16.201419 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:09:16.203776 11570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430315c358644870a30c3358c8e2a5a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41505 } }
I20260925 15:09:16.204093 11570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:16.204203 11570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430315c358644870a30c3358c8e2a5a6, State: Initialized, Role: FOLLOWER
I20260925 15:09:16.204684 11570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430315c358644870a30c3358c8e2a5a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41505 } }
I20260925 15:09:16.204892 11570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:16.204984 11570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:16.205103 11570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:16.207084 11570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430315c358644870a30c3358c8e2a5a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41505 } }
I20260925 15:09:16.207536 11570 leader_election.cc:302] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 430315c358644870a30c3358c8e2a5a6; no voters:
I20260925 15:09:16.208019 11570 leader_election.cc:288] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:16.208134 11580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:16.209110 11580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [term 1 LEADER]: Becoming Leader. State: Replica: 430315c358644870a30c3358c8e2a5a6, State: Running, Role: LEADER
I20260925 15:09:16.209707 11580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430315c358644870a30c3358c8e2a5a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41505 } }
I20260925 15:09:16.210261 11570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:16.212219 11581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "430315c358644870a30c3358c8e2a5a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430315c358644870a30c3358c8e2a5a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41505 } } }
I20260925 15:09:16.212631 11581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:16.213657 11582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 430315c358644870a30c3358c8e2a5a6. Latest consensus state: current_term: 1 leader_uuid: "430315c358644870a30c3358c8e2a5a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430315c358644870a30c3358c8e2a5a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41505 } } }
I20260925 15:09:16.213999 11588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:16.214028 11582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:16.220098 11588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:16.229182 11588 catalog_manager.cc:1347] Generated new cluster ID: f4999008ce3b40df805aab9591542521
I20260925 15:09:16.229290 11588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:16.237674 11588 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:16.239338 11588 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:16.250768 11588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 430315c358644870a30c3358c8e2a5a6: Generated new TSK 0
I20260925 15:09:16.251742 11588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:16.257822 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 11503
W20260925 15:09:16.258908 11565 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46985
W20260925 15:09:16.259583 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 11503: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:09:16.259778 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 11503
W20260925 15:09:16.273154 11488 connection.cc:535] client connection to 127.0.0.1:41505 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41505): Connection reset by peer (error 104)
2026-09-25T15:09:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1749 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:16 dist-test-slave-nsrc krb5kdc[11601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:16 dist-test-slave-nsrc krb5kdc[11601](info): set up 2 sockets
Sep 25 15:09:16 dist-test-slave-nsrc krb5kdc[11601](info): commencing operation
krb5kdc: starting...
W20260925 15:09:17.264420 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:17 dist-test-slave-nsrc krb5kdc[11601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:09:17.493341 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:17.800272 11614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:17.800567 11614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:17.800714 11614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:17.807735 11614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:17.807850 11614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:17.807934 11614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:17.808024 11614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:17.808082 11614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:17.817660 11614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39719
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:17.818905 11614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:17.820130 11614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:17.830941 11620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:17.831023 11619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:17.830984 11622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:17.831547 11614 server_base.cc:1034] running on GCE node
I20260925 15:09:17.832314 11614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:17.833683 11614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:17.835026 11614 hybrid_clock.cc:648] HybridClock initialized: now 1790348957834979 us; error 29 us; skew 500 ppm
I20260925 15:09:17.835355 11614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:17 dist-test-slave-nsrc krb5kdc[11601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:17.842830 11614 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:09:17.844471 11614 webserver.cc:460] Webserver started at http://127.0.0.1:44511/ using document root <none> and password file <none>
I20260925 15:09:17.845165 11614 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:17.845276 11614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:17.845566 11614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:17.848483 11614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "7dbe74fc77014c2e8bbbbf89e5a50983"
format_stamp: "Formatted at 2026-09-25 15:09:17 on dist-test-slave-nsrc"
server_key: "c66615b291d89c6ac9b45a3513268bff"
server_key_iv: "bab8378500480f939f6187d7a4dce4d9"
server_key_version: "encryptionkey@0"
I20260925 15:09:17.849215 11614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "7dbe74fc77014c2e8bbbbf89e5a50983"
format_stamp: "Formatted at 2026-09-25 15:09:17 on dist-test-slave-nsrc"
server_key: "c66615b291d89c6ac9b45a3513268bff"
server_key_iv: "bab8378500480f939f6187d7a4dce4d9"
server_key_version: "encryptionkey@0"
I20260925 15:09:17.854063 11614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 15:09:17.857280 11629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:17.858469 11614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260925 15:09:17.858614 11614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "7dbe74fc77014c2e8bbbbf89e5a50983"
format_stamp: "Formatted at 2026-09-25 15:09:17 on dist-test-slave-nsrc"
server_key: "c66615b291d89c6ac9b45a3513268bff"
server_key_iv: "bab8378500480f939f6187d7a4dce4d9"
server_key_version: "encryptionkey@0"
I20260925 15:09:17.858783 11614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:17.864029 11614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:17.867590 11614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:17.867842 11614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:17.884007 11614 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39719
I20260925 15:09:17.884051 11680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39719 every 8 connection(s)
I20260925 15:09:17.885355 11614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:17.888957 11681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:17.893460 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 11614
I20260925 15:09:17.893638 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:17.894193 23472 external_mini_cluster.cc:1428] Setting key ec4c3f98bbf2b640e39e701f390ca1d5
I20260925 15:09:17.899675 11681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983: Bootstrap starting.
I20260925 15:09:17.901448 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:17.895835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50409 (local address 127.0.0.1:39719)
0925 15:09:17.896578 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:17.896589 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:17.896629 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:09:17.898299 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:17.898319 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:17.898621 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:17.898923 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:17.899968 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:17.899974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:17.899986 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:17.900007 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:17.900061 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:17.900084 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:17.900294 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:17.900387 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 15:09:17.900546 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":331,"threads_started":1}
I20260925 15:09:17.902768 11681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:17.903798 11681 log.cc:824] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:17.904836 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:17.906675 11681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983: No bootstrap required, opened a new log
I20260925 15:09:17.911882 11684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:17.909803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46413 (local address 127.0.0.1:39719)
0925 15:09:17.909964 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:17.909969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:17.910168 (+ 199us) server_negotiation.cc:407] Connection header received
0925 15:09:17.910298 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:17.910303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:17.910361 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:17.910473 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:17.911264 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:17.911270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:17.911274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:17.911292 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:17.911338 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:17.911356 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:17.911476 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:17.911535 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 15:09:17.911588 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:09:17.915161 11681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe74fc77014c2e8bbbbf89e5a50983" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } }
I20260925 15:09:17.915470 11681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:17.915539 11681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dbe74fc77014c2e8bbbbf89e5a50983, State: Initialized, Role: FOLLOWER
I20260925 15:09:17.915990 11681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe74fc77014c2e8bbbbf89e5a50983" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } }
I20260925 15:09:17.916225 11681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:17.916337 11681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:17.916462 11681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:17.918503 11681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe74fc77014c2e8bbbbf89e5a50983" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } }
I20260925 15:09:17.918932 11681 leader_election.cc:302] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dbe74fc77014c2e8bbbbf89e5a50983; no voters:
I20260925 15:09:17.919428 11681 leader_election.cc:288] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:17.919552 11691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:17.920131 11691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [term 1 LEADER]: Becoming Leader. State: Replica: 7dbe74fc77014c2e8bbbbf89e5a50983, State: Running, Role: LEADER
I20260925 15:09:17.920682 11691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe74fc77014c2e8bbbbf89e5a50983" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } }
I20260925 15:09:17.921308 11681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:17.923115 11693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dbe74fc77014c2e8bbbbf89e5a50983. Latest consensus state: current_term: 1 leader_uuid: "7dbe74fc77014c2e8bbbbf89e5a50983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe74fc77014c2e8bbbbf89e5a50983" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } } }
I20260925 15:09:17.923885 11693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:17.923597 11692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dbe74fc77014c2e8bbbbf89e5a50983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe74fc77014c2e8bbbbf89e5a50983" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } } }
I20260925 15:09:17.924409 11692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:17.926208 11697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:17.931286 11697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:17.939815 11697 catalog_manager.cc:1347] Generated new cluster ID: c2a730fa390d49f0abebdc026d0fe699
I20260925 15:09:17.939934 11697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:17.950646 11697 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:17.951072 11697 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:17.957651 11697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dbe74fc77014c2e8bbbbf89e5a50983: Generated new TSK 0
I20260925 15:09:17.958580 11697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:17.969198 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 11614
W20260925 15:09:17.970278 11676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50409
W20260925 15:09:17.971035 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 11614: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:09:17.971185 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 11614
2026-09-25T15:09:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1710 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:18 dist-test-slave-nsrc krb5kdc[11712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:18 dist-test-slave-nsrc krb5kdc[11712](info): set up 2 sockets
Sep 25 15:09:18 dist-test-slave-nsrc krb5kdc[11712](info): commencing operation
krb5kdc: starting...
W20260925 15:09:18.968452 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:19 dist-test-slave-nsrc krb5kdc[11712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:09:19.199092 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:19.511193 11725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:19.511471 11725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:19.511567 11725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:19.519078 11725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:19.519191 11725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:19.519253 11725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:19.519311 11725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:19.519354 11725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:19.529291 11725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:19.530489 11725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:19.531700 11725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:19.542754 11731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:19.542793 11733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:19.542815 11730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:19.544306 11725 server_base.cc:1034] running on GCE node
I20260925 15:09:19.545101 11725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:19.546491 11725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:19.547690 11725 hybrid_clock.cc:648] HybridClock initialized: now 1790348959547635 us; error 51 us; skew 500 ppm
I20260925 15:09:19.548017 11725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:19 dist-test-slave-nsrc krb5kdc[11712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:19.550930 11725 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:09:19.552325 11725 webserver.cc:460] Webserver started at http://127.0.0.1:41007/ using document root <none> and password file <none>
I20260925 15:09:19.552968 11725 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:19.553079 11725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:19.553350 11725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:19.556306 11725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "31aca78b120441fd848caa069973c246"
format_stamp: "Formatted at 2026-09-25 15:09:19 on dist-test-slave-nsrc"
server_key: "5482a5adbb22acca59a37f0d4bd8959a"
server_key_iv: "6ad24bedec1329e78ba8c28f7eb5764e"
server_key_version: "encryptionkey@0"
I20260925 15:09:19.557021 11725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "31aca78b120441fd848caa069973c246"
format_stamp: "Formatted at 2026-09-25 15:09:19 on dist-test-slave-nsrc"
server_key: "5482a5adbb22acca59a37f0d4bd8959a"
server_key_iv: "6ad24bedec1329e78ba8c28f7eb5764e"
server_key_version: "encryptionkey@0"
I20260925 15:09:19.561753 11725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:09:19.565006 11740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:19.566205 11725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:09:19.566350 11725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "31aca78b120441fd848caa069973c246"
format_stamp: "Formatted at 2026-09-25 15:09:19 on dist-test-slave-nsrc"
server_key: "5482a5adbb22acca59a37f0d4bd8959a"
server_key_iv: "6ad24bedec1329e78ba8c28f7eb5764e"
server_key_version: "encryptionkey@0"
I20260925 15:09:19.566527 11725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:19.589672 11725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:19.593497 11725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:19.593765 11725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:19.610879 11725 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39337
I20260925 15:09:19.610914 11791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39337 every 8 connection(s)
I20260925 15:09:19.612547 11725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:19.616078 11792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:19.619714 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 11725
I20260925 15:09:19.619866 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:19.620329 23472 external_mini_cluster.cc:1428] Setting key 7ea88f87910886e07389552761f2bfb0
I20260925 15:09:19.628609 11792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246: Bootstrap starting.
I20260925 15:09:19.632340 11792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:19.633662 11792 log.cc:824] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:09:19 dist-test-slave-nsrc krb5kdc[11712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:09:19.637023 11792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246: No bootstrap required, opened a new log
I20260925 15:09:19.642993 11795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:19.622020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59739 (local address 127.0.0.1:39337)
0925 15:09:19.622754 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:19.622774 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:19.622813 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:09:19.624388 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:19.624411 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:19.624830 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:19.625157 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:19.627029 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:19.628492 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:19.630183 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:19.630845 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:19.630985 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:19.636005 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:19.636013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:19.636028 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:19.636062 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:19.639064 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:19.639862 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:19.639868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:19.639875 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:19.640032 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:19.640451 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:19.640455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:19.640457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:19.640947 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:19.641162 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:19.641629 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 15:09:19.641978 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":345,"threads_started":1}
I20260925 15:09:19.646107 11792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31aca78b120441fd848caa069973c246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260925 15:09:19.646508 11792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:19.646601 11792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31aca78b120441fd848caa069973c246, State: Initialized, Role: FOLLOWER
I20260925 15:09:19.646843 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:19.647166 11792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31aca78b120441fd848caa069973c246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260925 15:09:19.647394 11792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:19.647517 11792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:19.647657 11792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:19.649995 11792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31aca78b120441fd848caa069973c246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260925 15:09:19.650475 11792 leader_election.cc:302] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31aca78b120441fd848caa069973c246; no voters:
I20260925 15:09:19.651093 11792 leader_election.cc:288] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:19.651204 11801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:19.652451 11801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [term 1 LEADER]: Becoming Leader. State: Replica: 31aca78b120441fd848caa069973c246, State: Running, Role: LEADER
I20260925 15:09:19.652982 11801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31aca78b120441fd848caa069973c246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260925 15:09:19.653509 11792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:19.655412 11804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31aca78b120441fd848caa069973c246. Latest consensus state: current_term: 1 leader_uuid: "31aca78b120441fd848caa069973c246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31aca78b120441fd848caa069973c246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } } }
I20260925 15:09:19.655761 11804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:19.656297 11803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31aca78b120441fd848caa069973c246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31aca78b120441fd848caa069973c246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } } }
I20260925 15:09:19.656558 11803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:19.664770 11807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:19.669739 11807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:19.675871 11795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:19.651220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55537 (local address 127.0.0.1:39337)
0925 15:09:19.651386 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:19.651393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:19.657825 (+ 6432us) server_negotiation.cc:407] Connection header received
0925 15:09:19.657965 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:19.657971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:19.658058 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:19.658176 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:19.660197 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:19.661392 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:19.663274 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:19.663987 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:19.664108 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:19.667205 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:19.667213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:19.667219 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:19.667261 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:19.672001 (+ 4740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:19.672903 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:19.672909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:19.672912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:19.673058 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:19.673492 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:19.673497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:19.673500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:19.673921 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:19.674160 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:19.675342 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0925 15:09:19.675539 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:09:19.680346 11807 catalog_manager.cc:1347] Generated new cluster ID: d1bdd4824e3940029b1940a73260af0b
I20260925 15:09:19.680445 11807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:19.688536 11807 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:19.689874 11807 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:19.696945 11807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31aca78b120441fd848caa069973c246: Generated new TSK 0
I20260925 15:09:19.697710 11807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:19.708261 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 11725
I20260925 15:09:19.709218 11787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:19.797330 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 11725
2026-09-25T15:09:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1826 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:19 dist-test-slave-nsrc krb5kdc[11824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:19 dist-test-slave-nsrc krb5kdc[11824](info): set up 2 sockets
Sep 25 15:09:19 dist-test-slave-nsrc krb5kdc[11824](info): commencing operation
krb5kdc: starting...
W20260925 15:09:20.808432 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:20 dist-test-slave-nsrc krb5kdc[11824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:09:21.035017 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:21.350868 11837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:21.351135 11837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:21.351222 11837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:21.358361 11837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:21.358469 11837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:21.358521 11837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:21.358582 11837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:21.358623 11837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:21.368183 11837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:21.369387 11837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:21.370604 11837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:21.381989 11842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:21.381902 11843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:21.381973 11845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:21.383015 11837 server_base.cc:1034] running on GCE node
I20260925 15:09:21.383822 11837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:21.385356 11837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:21.386590 11837 hybrid_clock.cc:648] HybridClock initialized: now 1790348961386541 us; error 49 us; skew 500 ppm
I20260925 15:09:21.386934 11837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:21 dist-test-slave-nsrc krb5kdc[11824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:21.389935 11837 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:09:21.391319 11837 webserver.cc:460] Webserver started at http://127.0.0.1:35569/ using document root <none> and password file <none>
I20260925 15:09:21.391976 11837 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:21.392093 11837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:21.392406 11837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:21.395371 11837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "0ce7af41213049fdbffd5ce6481c9de7"
format_stamp: "Formatted at 2026-09-25 15:09:21 on dist-test-slave-nsrc"
server_key: "4b9b4e58ddf44ac941bc9d2494e6c815"
server_key_iv: "5ed827a4b9775944657e7e1819587f45"
server_key_version: "encryptionkey@0"
I20260925 15:09:21.396106 11837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "0ce7af41213049fdbffd5ce6481c9de7"
format_stamp: "Formatted at 2026-09-25 15:09:21 on dist-test-slave-nsrc"
server_key: "4b9b4e58ddf44ac941bc9d2494e6c815"
server_key_iv: "5ed827a4b9775944657e7e1819587f45"
server_key_version: "encryptionkey@0"
I20260925 15:09:21.400900 11837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:09:21.403970 11852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:21.405215 11837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:09:21.405365 11837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "0ce7af41213049fdbffd5ce6481c9de7"
format_stamp: "Formatted at 2026-09-25 15:09:21 on dist-test-slave-nsrc"
server_key: "4b9b4e58ddf44ac941bc9d2494e6c815"
server_key_iv: "5ed827a4b9775944657e7e1819587f45"
server_key_version: "encryptionkey@0"
I20260925 15:09:21.405539 11837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:21.422960 11837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:21.427199 11837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:21.427466 11837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:21.443724 11837 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39515
I20260925 15:09:21.443737 11903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39515 every 8 connection(s)
I20260925 15:09:21.445097 11837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:21.446647 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 11837
I20260925 15:09:21.446785 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:21.447190 23472 external_mini_cluster.cc:1428] Setting key 61b16472f7de60e36b96b70ebecce23f
I20260925 15:09:21.448958 11904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:21.459513 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:21.448911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38261 (local address 127.0.0.1:39515)
0925 15:09:21.449629 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:21.449639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:21.449678 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:09:21.451186 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:21.451206 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:21.451582 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:21.451902 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:21.452429 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:21.453686 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:21.455430 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:21.456063 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:21.456236 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:21.457381 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:21.457389 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:21.457405 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:21.457922 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:21.457995 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:21.458022 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:21.458114 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:21.458341 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 15:09:21.458584 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":316,"threads_started":1}
I20260925 15:09:21.460908 11904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7: Bootstrap starting.
I20260925 15:09:21.463186 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:21.464030 11904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:21.465015 11904 log.cc:824] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:21.467852 11904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7: No bootstrap required, opened a new log
I20260925 15:09:21.474555 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:21.468366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56119 (local address 127.0.0.1:39515)
0925 15:09:21.468541 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:21.468548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:21.468922 (+ 374us) server_negotiation.cc:407] Connection header received
0925 15:09:21.469087 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:21.469091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:21.469151 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:21.469225 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:21.469827 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:21.470547 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:21.472298 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:21.472744 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:21.472836 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:21.473909 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:21.473913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:21.473916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:21.473932 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:21.473969 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:21.473984 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:21.474050 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:21.474187 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:09:21.474253 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:09:21.476962 11904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce7af41213049fdbffd5ce6481c9de7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } }
I20260925 15:09:21.477303 11904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:21.477391 11904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce7af41213049fdbffd5ce6481c9de7, State: Initialized, Role: FOLLOWER
I20260925 15:09:21.477918 11904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce7af41213049fdbffd5ce6481c9de7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } }
I20260925 15:09:21.478153 11904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:21.478276 11904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:21.478437 11904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:21.480880 11904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce7af41213049fdbffd5ce6481c9de7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } }
I20260925 15:09:21.481448 11904 leader_election.cc:302] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ce7af41213049fdbffd5ce6481c9de7; no voters:
I20260925 15:09:21.482108 11904 leader_election.cc:288] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:21.482239 11914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:21.483762 11914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [term 1 LEADER]: Becoming Leader. State: Replica: 0ce7af41213049fdbffd5ce6481c9de7, State: Running, Role: LEADER
I20260925 15:09:21.484306 11914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce7af41213049fdbffd5ce6481c9de7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } }
I20260925 15:09:21.484562 11904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:21.486795 11915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ce7af41213049fdbffd5ce6481c9de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce7af41213049fdbffd5ce6481c9de7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } } }
I20260925 15:09:21.487138 11915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:21.488374 11916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ce7af41213049fdbffd5ce6481c9de7. Latest consensus state: current_term: 1 leader_uuid: "0ce7af41213049fdbffd5ce6481c9de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce7af41213049fdbffd5ce6481c9de7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } } }
I20260925 15:09:21.488692 11916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:21.489257 11922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:21.494395 11922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:21.502503 11922 catalog_manager.cc:1347] Generated new cluster ID: 3df0bb9cf8c84c94b916c72baa748f00
I20260925 15:09:21.502599 11922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:21.511446 11922 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:21.513183 11922 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:21.524813 11922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ce7af41213049fdbffd5ce6481c9de7: Generated new TSK 0
I20260925 15:09:21.525826 11922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:21.542006 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 11837
W20260925 15:09:21.543046 11899 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38261
W20260925 15:09:21.543699 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 11837: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:09:21.543809 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 11837
2026-09-25T15:09:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1746 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:21 dist-test-slave-nsrc krb5kdc[11935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:21 dist-test-slave-nsrc krb5kdc[11935](info): set up 2 sockets
Sep 25 15:09:21 dist-test-slave-nsrc krb5kdc[11935](info): commencing operation
krb5kdc: starting...
W20260925 15:09:22.548357 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:22 dist-test-slave-nsrc krb5kdc[11935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:09:22.773691 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33399
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:23.083863 11948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:23.084246 11948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:23.084400 11948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:23.094377 11948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:23.094517 11948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:23.094591 11948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:23.094676 11948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:23.094738 11948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:23.105499 11948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:23.106722 11948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:23.107990 11948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:23.118966 11953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:23.119074 11954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:23.119004 11956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:23.120045 11948 server_base.cc:1034] running on GCE node
I20260925 15:09:23.120812 11948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:23.122133 11948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:23.123323 11948 hybrid_clock.cc:648] HybridClock initialized: now 1790348963123286 us; error 27 us; skew 500 ppm
I20260925 15:09:23.123643 11948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:23 dist-test-slave-nsrc krb5kdc[11935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:23.131795 11948 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:09:23.133275 11948 webserver.cc:460] Webserver started at http://127.0.0.1:33655/ using document root <none> and password file <none>
I20260925 15:09:23.133895 11948 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:23.134011 11948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:23.134289 11948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:23.137357 11948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "5cbbb6fbab874312b38cbd278c0bcc84"
format_stamp: "Formatted at 2026-09-25 15:09:23 on dist-test-slave-nsrc"
server_key: "0206fae508350d2ad39f2518767986ca"
server_key_iv: "0eb0ac2690612cc76cbe17b94fb5d62f"
server_key_version: "encryptionkey@0"
I20260925 15:09:23.138175 11948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "5cbbb6fbab874312b38cbd278c0bcc84"
format_stamp: "Formatted at 2026-09-25 15:09:23 on dist-test-slave-nsrc"
server_key: "0206fae508350d2ad39f2518767986ca"
server_key_iv: "0eb0ac2690612cc76cbe17b94fb5d62f"
server_key_version: "encryptionkey@0"
I20260925 15:09:23.143568 11948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:09:23.147143 11963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:23.148484 11948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:23.148643 11948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "5cbbb6fbab874312b38cbd278c0bcc84"
format_stamp: "Formatted at 2026-09-25 15:09:23 on dist-test-slave-nsrc"
server_key: "0206fae508350d2ad39f2518767986ca"
server_key_iv: "0eb0ac2690612cc76cbe17b94fb5d62f"
server_key_version: "encryptionkey@0"
I20260925 15:09:23.148818 11948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:23.154265 11948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:23.157979 11948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:23.158227 11948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:23.174836 11948 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33399
I20260925 15:09:23.174842 12014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33399 every 8 connection(s)
I20260925 15:09:23.176213 11948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:23.179667 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:23.185142 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 11948
I20260925 15:09:23.185283 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:23.185678 23472 external_mini_cluster.cc:1428] Setting key 282cd0cf221f2700f9b50f325c53ace0
I20260925 15:09:23.189793 12015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84: Bootstrap starting.
I20260925 15:09:23.193053 12018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:23.187349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51003 (local address 127.0.0.1:33399)
0925 15:09:23.188187 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:23.188201 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:23.188242 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:09:23.189814 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:23.189835 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:23.190189 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:23.190481 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:23.191522 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:23.191528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:23.191541 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:23.191554 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:23.191619 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:23.191638 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:23.191797 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:23.191956 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:09:23.192131 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":351,"threads_started":1}
I20260925 15:09:23.193137 12015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:23.194149 12015 log.cc:824] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:23.196674 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:23.197109 12015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84: No bootstrap required, opened a new log
I20260925 15:09:23.202903 12018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:23.200644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59717 (local address 127.0.0.1:33399)
0925 15:09:23.200789 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:23.200795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:23.201166 (+ 371us) server_negotiation.cc:407] Connection header received
0925 15:09:23.201317 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:23.201321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:23.201364 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:23.201445 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:23.202299 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:23.202304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:23.202309 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:23.202326 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:23.202366 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:23.202382 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:23.202506 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:23.202582 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 15:09:23.202636 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:09:23.205088 12015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cbbb6fbab874312b38cbd278c0bcc84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33399 } }
I20260925 15:09:23.205425 12015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:23.205523 12015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbbb6fbab874312b38cbd278c0bcc84, State: Initialized, Role: FOLLOWER
I20260925 15:09:23.206079 12015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cbbb6fbab874312b38cbd278c0bcc84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33399 } }
I20260925 15:09:23.206300 12015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:23.206444 12015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:23.206604 12015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:23.208904 12015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cbbb6fbab874312b38cbd278c0bcc84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33399 } }
I20260925 15:09:23.209395 12015 leader_election.cc:302] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cbbb6fbab874312b38cbd278c0bcc84; no voters:
I20260925 15:09:23.210045 12015 leader_election.cc:288] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:23.210173 12025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:23.210788 12025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [term 1 LEADER]: Becoming Leader. State: Replica: 5cbbb6fbab874312b38cbd278c0bcc84, State: Running, Role: LEADER
I20260925 15:09:23.211256 12025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cbbb6fbab874312b38cbd278c0bcc84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33399 } }
I20260925 15:09:23.211938 12015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:23.213936 12027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cbbb6fbab874312b38cbd278c0bcc84. Latest consensus state: current_term: 1 leader_uuid: "5cbbb6fbab874312b38cbd278c0bcc84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cbbb6fbab874312b38cbd278c0bcc84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33399 } } }
I20260925 15:09:23.214367 12027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:23.214035 12026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cbbb6fbab874312b38cbd278c0bcc84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cbbb6fbab874312b38cbd278c0bcc84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33399 } } }
I20260925 15:09:23.215075 12026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:23.216025 12033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:23.221202 12033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:23.229485 12033 catalog_manager.cc:1347] Generated new cluster ID: 6e7f561bd85f4a779d7ae798f4c5c7fe
I20260925 15:09:23.229593 12033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:23.250845 12033 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:23.251255 12033 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:23.261407 12033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cbbb6fbab874312b38cbd278c0bcc84: Generated new TSK 0
I20260925 15:09:23.262167 12033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:23.271129 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 11948
W20260925 15:09:23.272222 12010 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51003
W20260925 15:09:23.272900 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 11948: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:09:23.273049 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 11948
2026-09-25T15:09:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1729 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:23 dist-test-slave-nsrc krb5kdc[12046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:23 dist-test-slave-nsrc krb5kdc[12046](info): set up 2 sockets
Sep 25 15:09:23 dist-test-slave-nsrc krb5kdc[12046](info): commencing operation
krb5kdc: starting...
W20260925 15:09:24.288465 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:24 dist-test-slave-nsrc krb5kdc[12046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:09:24.521306 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:24.834890 12059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:24.835175 12059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:24.835307 12059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:24.842772 12059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:24.842900 12059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:24.842983 12059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:24.843071 12059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:24.843132 12059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:24.853048 12059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:24.854460 12059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:24.855753 12059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:24.866993 12065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:24.867033 12067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:24.867216 12064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:24.868057 12059 server_base.cc:1034] running on GCE node
I20260925 15:09:24.868872 12059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:24.870242 12059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:24.871455 12059 hybrid_clock.cc:648] HybridClock initialized: now 1790348964871408 us; error 42 us; skew 500 ppm
I20260925 15:09:24.871793 12059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:24 dist-test-slave-nsrc krb5kdc[12046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:24.874806 12059 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:09:24.876260 12059 webserver.cc:460] Webserver started at http://127.0.0.1:39537/ using document root <none> and password file <none>
I20260925 15:09:24.876904 12059 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:24.877018 12059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:24.877292 12059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:24.880095 12059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "9192220a686648868c0e157173b77aa8"
format_stamp: "Formatted at 2026-09-25 15:09:24 on dist-test-slave-nsrc"
server_key: "224e30e682b0d78ae14306aa58a5ef0f"
server_key_iv: "e84779eec0506d6c9967abfaa576b094"
server_key_version: "encryptionkey@0"
I20260925 15:09:24.880858 12059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "9192220a686648868c0e157173b77aa8"
format_stamp: "Formatted at 2026-09-25 15:09:24 on dist-test-slave-nsrc"
server_key: "224e30e682b0d78ae14306aa58a5ef0f"
server_key_iv: "e84779eec0506d6c9967abfaa576b094"
server_key_version: "encryptionkey@0"
I20260925 15:09:24.885550 12059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:09:24.888700 12074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:24.889957 12059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:24.890110 12059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "9192220a686648868c0e157173b77aa8"
format_stamp: "Formatted at 2026-09-25 15:09:24 on dist-test-slave-nsrc"
server_key: "224e30e682b0d78ae14306aa58a5ef0f"
server_key_iv: "e84779eec0506d6c9967abfaa576b094"
server_key_version: "encryptionkey@0"
I20260925 15:09:24.890280 12059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:24.938658 12059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:24.942790 12059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:24.943058 12059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:24.959878 12059 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34903
I20260925 15:09:24.959915 12125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34903 every 8 connection(s)
I20260925 15:09:24.961220 12059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:24.962471 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 12059
I20260925 15:09:24.962646 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:24.963213 23472 external_mini_cluster.cc:1428] Setting key 08641acca89afda0cb692c80728fc525
I20260925 15:09:24.965556 12126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:09:24 dist-test-slave-nsrc krb5kdc[12046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:09:24.977838 12126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8: Bootstrap starting.
I20260925 15:09:24.980938 12126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:24.981949 12126 log.cc:824] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:24.985201 12126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8: No bootstrap required, opened a new log
I20260925 15:09:24.985563 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:24.965023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:46569 (local address 127.0.0.1:34903)
0925 15:09:24.965732 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:24.965746 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:24.965785 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:09:24.967367 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:24.967387 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:24.967756 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:24.968076 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:24.969831 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:24.971131 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:24.972850 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:24.973485 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:24.973602 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:24.978193 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:24.978200 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:24.978214 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:24.978248 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:24.981172 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:24.981956 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:24.981962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:24.981969 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:24.982146 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:24.982655 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:24.982659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:24.982661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:24.983228 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:24.983527 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:24.983958 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 15:09:24.984300 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":320,"threads_started":1}
I20260925 15:09:24.989198 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:24.994490 12126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9192220a686648868c0e157173b77aa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34903 } }
I20260925 15:09:24.994871 12126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:24.994964 12126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9192220a686648868c0e157173b77aa8, State: Initialized, Role: FOLLOWER
I20260925 15:09:24.995509 12126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9192220a686648868c0e157173b77aa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34903 } }
I20260925 15:09:24.995718 12126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:24.995847 12126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:24.996011 12126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:24.998132 12126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9192220a686648868c0e157173b77aa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34903 } }
I20260925 15:09:24.998600 12126 leader_election.cc:302] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9192220a686648868c0e157173b77aa8; no voters:
I20260925 15:09:24.999187 12126 leader_election.cc:288] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:24.999289 12136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:25.000463 12136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [term 1 LEADER]: Becoming Leader. State: Replica: 9192220a686648868c0e157173b77aa8, State: Running, Role: LEADER
I20260925 15:09:25.001096 12136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9192220a686648868c0e157173b77aa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34903 } }
I20260925 15:09:25.001485 12126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:25.003975 12137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9192220a686648868c0e157173b77aa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9192220a686648868c0e157173b77aa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34903 } } }
I20260925 15:09:25.003991 12138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9192220a686648868c0e157173b77aa8. Latest consensus state: current_term: 1 leader_uuid: "9192220a686648868c0e157173b77aa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9192220a686648868c0e157173b77aa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34903 } } }
I20260925 15:09:25.004383 12137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:25.004383 12138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:25.009418 12142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:25.015200 12142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:25.019346 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:24.994229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:45889 (local address 127.0.0.1:34903)
0925 15:09:24.994384 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:24.994389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:25.000769 (+ 6380us) server_negotiation.cc:407] Connection header received
0925 15:09:25.000957 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:25.000962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:25.001042 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:25.001157 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:25.002775 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:25.004129 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:25.006243 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:25.006931 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:25.007038 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:25.010500 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:25.010507 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:25.010512 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:25.010560 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:25.014694 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:25.015764 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:25.015787 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:25.015791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:25.016067 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:25.016693 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:25.016701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:25.016705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:25.017119 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:25.018564 (+ 1445us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:25.018752 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 15:09:25.018963 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:09:25.025010 12142 catalog_manager.cc:1347] Generated new cluster ID: b616bfd318ac45a7bcc04cab7245a84d
I20260925 15:09:25.025174 12142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:25.053025 12142 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:25.054272 12142 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:25.063557 12142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9192220a686648868c0e157173b77aa8: Generated new TSK 0
I20260925 15:09:25.064350 12142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:25.072517 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12059
I20260925 15:09:25.074177 12121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:25.167135 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 12059
W20260925 15:09:25.180761 12044 connection.cc:535] client connection to 127.0.0.1:34903 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34903): Connection reset by peer (error 104)
2026-09-25T15:09:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1894 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:25 dist-test-slave-nsrc krb5kdc[12158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:25 dist-test-slave-nsrc krb5kdc[12158](info): set up 2 sockets
Sep 25 15:09:25 dist-test-slave-nsrc krb5kdc[12158](info): commencing operation
krb5kdc: starting...
W20260925 15:09:26.180562 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:26 dist-test-slave-nsrc krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:09:26.410588 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:26.725622 12171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:26.725932 12171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:26.726073 12171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:26.733218 12171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:26.733340 12171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:26.733419 12171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:26.733505 12171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:26.733561 12171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:26.743752 12171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:26.745075 12171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:26.746323 12171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:26.757169 12176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:26.757285 12179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:26.757392 12177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:26.758592 12171 server_base.cc:1034] running on GCE node
I20260925 15:09:26.759341 12171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:26.760820 12171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:26.762017 12171 hybrid_clock.cc:648] HybridClock initialized: now 1790348966761973 us; error 35 us; skew 500 ppm
I20260925 15:09:26.762331 12171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:26 dist-test-slave-nsrc krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:26.765317 12171 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:09:26.766680 12171 webserver.cc:460] Webserver started at http://127.0.0.1:41725/ using document root <none> and password file <none>
I20260925 15:09:26.767298 12171 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:26.767417 12171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:26.767686 12171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:26.770642 12171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "ec0633a64a2b4cff95fb2ef621d06479"
format_stamp: "Formatted at 2026-09-25 15:09:26 on dist-test-slave-nsrc"
server_key: "9a7b8c1f71e6bc9cb545a1f226c3b3a5"
server_key_iv: "8c082663e58527deb933d4d420821eb5"
server_key_version: "encryptionkey@0"
I20260925 15:09:26.771341 12171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "ec0633a64a2b4cff95fb2ef621d06479"
format_stamp: "Formatted at 2026-09-25 15:09:26 on dist-test-slave-nsrc"
server_key: "9a7b8c1f71e6bc9cb545a1f226c3b3a5"
server_key_iv: "8c082663e58527deb933d4d420821eb5"
server_key_version: "encryptionkey@0"
I20260925 15:09:26.776073 12171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 15:09:26.779285 12186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:26.780620 12171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 15:09:26.780769 12171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "ec0633a64a2b4cff95fb2ef621d06479"
format_stamp: "Formatted at 2026-09-25 15:09:26 on dist-test-slave-nsrc"
server_key: "9a7b8c1f71e6bc9cb545a1f226c3b3a5"
server_key_iv: "8c082663e58527deb933d4d420821eb5"
server_key_version: "encryptionkey@0"
I20260925 15:09:26.780953 12171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:26.805218 12171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:26.809608 12171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:26.809894 12171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:26.826741 12171 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40257
I20260925 15:09:26.826764 12237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40257 every 8 connection(s)
I20260925 15:09:26.828017 12171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:26.830869 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 12171
I20260925 15:09:26.831022 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:26.831477 23472 external_mini_cluster.cc:1428] Setting key b051a6355bcc96b69f6f8bd80ce9998f
I20260925 15:09:26.831580 12238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:26.844437 12238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479: Bootstrap starting.
I20260925 15:09:26.844602 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:26.833102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:60421 (local address 127.0.0.1:40257)
0925 15:09:26.833861 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:26.833872 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:26.833913 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:09:26.835879 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:26.835899 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:26.836332 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:26.836641 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:26.837242 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:26.838568 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:26.840302 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:26.840981 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:26.841118 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:26.842403 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:26.842426 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:26.842442 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:26.842957 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:26.843038 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:26.843061 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:26.843183 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:26.843475 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:09:26.843632 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":331,"threads_started":1}
I20260925 15:09:26.847836 12238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:26.848235 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:26.848860 12238 log.cc:824] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:26.851491 12238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479: No bootstrap required, opened a new log
I20260925 15:09:26.858741 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:26.852358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:34267 (local address 127.0.0.1:40257)
0925 15:09:26.852526 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:26.852531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:26.853033 (+ 502us) server_negotiation.cc:407] Connection header received
0925 15:09:26.853171 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:26.853179 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:26.853264 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:26.853368 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:26.853822 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:26.854950 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:26.856238 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:26.856928 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:26.857069 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:26.858017 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:26.858023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:26.858027 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:26.858047 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:26.858094 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:26.858113 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:26.858233 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:26.858360 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 15:09:26.858428 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:09:26.860589 12238 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0633a64a2b4cff95fb2ef621d06479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40257 } }
I20260925 15:09:26.860942 12238 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:26.861033 12238 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0633a64a2b4cff95fb2ef621d06479, State: Initialized, Role: FOLLOWER
I20260925 15:09:26.861604 12238 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0633a64a2b4cff95fb2ef621d06479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40257 } }
I20260925 15:09:26.861883 12238 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:26.862023 12238 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:26.862193 12238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:26.864341 12238 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0633a64a2b4cff95fb2ef621d06479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40257 } }
I20260925 15:09:26.864812 12238 leader_election.cc:302] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec0633a64a2b4cff95fb2ef621d06479; no voters:
I20260925 15:09:26.865326 12238 leader_election.cc:288] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:26.865433 12248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:26.866448 12248 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [term 1 LEADER]: Becoming Leader. State: Replica: ec0633a64a2b4cff95fb2ef621d06479, State: Running, Role: LEADER
I20260925 15:09:26.867039 12248 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0633a64a2b4cff95fb2ef621d06479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40257 } }
I20260925 15:09:26.867309 12238 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:26.869329 12250 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec0633a64a2b4cff95fb2ef621d06479. Latest consensus state: current_term: 1 leader_uuid: "ec0633a64a2b4cff95fb2ef621d06479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0633a64a2b4cff95fb2ef621d06479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40257 } } }
I20260925 15:09:26.869421 12249 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec0633a64a2b4cff95fb2ef621d06479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0633a64a2b4cff95fb2ef621d06479" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40257 } } }
I20260925 15:09:26.869647 12250 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:26.869765 12249 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:26.870898 12254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:26.877228 12254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:26.885787 12254 catalog_manager.cc:1347] Generated new cluster ID: c762d9407e604242a2d73f4557901d3c
I20260925 15:09:26.885902 12254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:26.921322 12254 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:26.922811 12254 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:26.928755 12254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec0633a64a2b4cff95fb2ef621d06479: Generated new TSK 0
I20260925 15:09:26.929524 12254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:26.940459 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12171
W20260925 15:09:26.941550 12233 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.53:60421
W20260925 15:09:26.942231 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12171: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:09:26.942380 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 12171
W20260925 15:09:26.955591 12156 connection.cc:535] client connection to 127.0.0.1:40257 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40257): Connection reset by peer (error 104)
2026-09-25T15:09:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1774 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:27 dist-test-slave-nsrc krb5kdc[12269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:27 dist-test-slave-nsrc krb5kdc[12269](info): set up 2 sockets
Sep 25 15:09:27 dist-test-slave-nsrc krb5kdc[12269](info): commencing operation
krb5kdc: starting...
W20260925 15:09:27.952378 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:28 dist-test-slave-nsrc krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:09:28.177217 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43109
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:28.486281 12282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:28.486552 12282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:28.486685 12282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:28.493815 12282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:28.493945 12282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:28.494027 12282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:28.494107 12282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:28.494166 12282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:28.503809 12282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:28.505409 12282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:28.507072 12282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:28.518059 12287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:28.518016 12288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:28.518100 12290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:28.518887 12282 server_base.cc:1034] running on GCE node
I20260925 15:09:28.519560 12282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:28.520882 12282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:28.522078 12282 hybrid_clock.cc:648] HybridClock initialized: now 1790348968522040 us; error 30 us; skew 500 ppm
I20260925 15:09:28.522387 12282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:28 dist-test-slave-nsrc krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:28.529708 12282 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:09:28.531172 12282 webserver.cc:460] Webserver started at http://127.0.0.1:40153/ using document root <none> and password file <none>
I20260925 15:09:28.531809 12282 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:28.531931 12282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:28.532267 12282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:28.535169 12282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "6c7d75752cc24308ba672404f7be39c2"
format_stamp: "Formatted at 2026-09-25 15:09:28 on dist-test-slave-nsrc"
server_key: "b2962920bb72676cb1357adaeb1fef4d"
server_key_iv: "9ebef5f7d357d636ff7baec5d89c3be9"
server_key_version: "encryptionkey@0"
I20260925 15:09:28.535910 12282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "6c7d75752cc24308ba672404f7be39c2"
format_stamp: "Formatted at 2026-09-25 15:09:28 on dist-test-slave-nsrc"
server_key: "b2962920bb72676cb1357adaeb1fef4d"
server_key_iv: "9ebef5f7d357d636ff7baec5d89c3be9"
server_key_version: "encryptionkey@0"
I20260925 15:09:28.540850 12282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 15:09:28.544039 12297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:28.545312 12282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:09:28.545463 12282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "6c7d75752cc24308ba672404f7be39c2"
format_stamp: "Formatted at 2026-09-25 15:09:28 on dist-test-slave-nsrc"
server_key: "b2962920bb72676cb1357adaeb1fef4d"
server_key_iv: "9ebef5f7d357d636ff7baec5d89c3be9"
server_key_version: "encryptionkey@0"
I20260925 15:09:28.545639 12282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:28.551072 12282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:28.554733 12282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:28.554992 12282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:28.571147 12282 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43109
I20260925 15:09:28.571182 12348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43109 every 8 connection(s)
I20260925 15:09:28.572470 12282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:28.575959 12349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:28.577540 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 12282
I20260925 15:09:28.577708 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:28.578209 23472 external_mini_cluster.cc:1428] Setting key 98bc030a91584d469b1f50f0c135c567
I20260925 15:09:28.585458 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:28.579993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:46689 (local address 127.0.0.1:43109)
0925 15:09:28.580718 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:28.580729 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:28.580776 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:09:28.582281 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:28.582302 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:28.582635 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:28.582993 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:28.583957 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:28.583963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:28.583975 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:28.583989 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:28.584045 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:28.584067 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:28.584289 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:28.584411 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 15:09:28.584581 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":320,"threads_started":1}
I20260925 15:09:28.588542 12349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2: Bootstrap starting.
I20260925 15:09:28.588876 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:28.591280 12349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:28.592372 12349 log.cc:824] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:28.595153 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:28.592967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:49845 (local address 127.0.0.1:43109)
0925 15:09:28.593143 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:28.593148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:28.593487 (+ 339us) server_negotiation.cc:407] Connection header received
0925 15:09:28.593655 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:28.593660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:28.593713 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:28.593814 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:28.594597 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:28.594602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:28.594606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:28.594622 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:28.594664 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:28.594681 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:28.594796 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:28.594839 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 15:09:28.594905 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:09:28.595320 12349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2: No bootstrap required, opened a new log
I20260925 15:09:28.602972 12349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7d75752cc24308ba672404f7be39c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43109 } }
I20260925 15:09:28.603413 12349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:28.603509 12349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7d75752cc24308ba672404f7be39c2, State: Initialized, Role: FOLLOWER
I20260925 15:09:28.604019 12349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7d75752cc24308ba672404f7be39c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43109 } }
I20260925 15:09:28.604251 12349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:28.604373 12349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:28.604516 12349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:28.606575 12349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7d75752cc24308ba672404f7be39c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43109 } }
I20260925 15:09:28.607041 12349 leader_election.cc:302] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c7d75752cc24308ba672404f7be39c2; no voters:
I20260925 15:09:28.607553 12349 leader_election.cc:288] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:28.607661 12359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:28.608266 12359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [term 1 LEADER]: Becoming Leader. State: Replica: 6c7d75752cc24308ba672404f7be39c2, State: Running, Role: LEADER
I20260925 15:09:28.608762 12359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7d75752cc24308ba672404f7be39c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43109 } }
I20260925 15:09:28.609460 12349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:28.611579 12360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c7d75752cc24308ba672404f7be39c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7d75752cc24308ba672404f7be39c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43109 } } }
I20260925 15:09:28.612398 12360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:28.612025 12361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c7d75752cc24308ba672404f7be39c2. Latest consensus state: current_term: 1 leader_uuid: "6c7d75752cc24308ba672404f7be39c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7d75752cc24308ba672404f7be39c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43109 } } }
I20260925 15:09:28.612841 12361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:28.613310 12368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:28.618485 12368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:28.627142 12368 catalog_manager.cc:1347] Generated new cluster ID: 64a7330fad144261bffc569f4838e4e8
I20260925 15:09:28.627251 12368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:28.643245 12368 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:28.643610 12368 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:28.652832 12368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c7d75752cc24308ba672404f7be39c2: Generated new TSK 0
I20260925 15:09:28.653571 12368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:28.664534 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12282
W20260925 15:09:28.665550 12344 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.53:46689
W20260925 15:09:28.666208 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12282: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:09:28.666329 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 12282
2026-09-25T15:09:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1723 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:28 dist-test-slave-nsrc krb5kdc[12380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:28 dist-test-slave-nsrc krb5kdc[12380](info): set up 2 sockets
Sep 25 15:09:28 dist-test-slave-nsrc krb5kdc[12380](info): commencing operation
krb5kdc: starting...
W20260925 15:09:29.672438 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:29 dist-test-slave-nsrc krb5kdc[12380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:09:29.903301 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:30.221138 12393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:30.221452 12393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:30.221601 12393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:30.229771 12393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:30.229902 12393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:30.229978 12393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:30.230058 12393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:30.230118 12393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:30.240135 12393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40167
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:30.241485 12393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:30.242779 12393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:30.254627 12401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:30.254719 12399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:30.254659 12398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:30.255390 12393 server_base.cc:1034] running on GCE node
I20260925 15:09:30.256181 12393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:30.257658 12393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:30.258868 12393 hybrid_clock.cc:648] HybridClock initialized: now 1790348970258822 us; error 34 us; skew 500 ppm
I20260925 15:09:30.259209 12393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:30 dist-test-slave-nsrc krb5kdc[12380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:30.262297 12393 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:09:30.263767 12393 webserver.cc:460] Webserver started at http://127.0.0.1:46269/ using document root <none> and password file <none>
I20260925 15:09:30.264555 12393 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:30.264676 12393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:30.264999 12393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:30.268054 12393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "67b1157be78b4ba492f092c8c78a685f"
format_stamp: "Formatted at 2026-09-25 15:09:30 on dist-test-slave-nsrc"
server_key: "e19f1eaa45a59615f0ac4b397450a184"
server_key_iv: "a5901f87f98f65ec94ac57c42f064efd"
server_key_version: "encryptionkey@0"
I20260925 15:09:30.268864 12393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "67b1157be78b4ba492f092c8c78a685f"
format_stamp: "Formatted at 2026-09-25 15:09:30 on dist-test-slave-nsrc"
server_key: "e19f1eaa45a59615f0ac4b397450a184"
server_key_iv: "a5901f87f98f65ec94ac57c42f064efd"
server_key_version: "encryptionkey@0"
I20260925 15:09:30.273976 12393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:09:30.277493 12408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:30.278806 12393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:09:30.278971 12393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "67b1157be78b4ba492f092c8c78a685f"
format_stamp: "Formatted at 2026-09-25 15:09:30 on dist-test-slave-nsrc"
server_key: "e19f1eaa45a59615f0ac4b397450a184"
server_key_iv: "a5901f87f98f65ec94ac57c42f064efd"
server_key_version: "encryptionkey@0"
I20260925 15:09:30.279192 12393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:30.297560 12393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:30.301795 12393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:30.302088 12393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:30.319892 12393 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40167
I20260925 15:09:30.319928 12459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40167 every 8 connection(s)
I20260925 15:09:30.321288 12393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:30.325138 12460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:30.325246 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 12393
I20260925 15:09:30.325420 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:30.325860 23472 external_mini_cluster.cc:1428] Setting key cbb534806f8fbc3fda8661135e7a8bae
I20260925 15:09:30.339054 12460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f: Bootstrap starting.
Sep 25 15:09:30 dist-test-slave-nsrc krb5kdc[12380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:09:30.342387 12460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:30.343515 12460 log.cc:824] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:30.346297 12460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f: No bootstrap required, opened a new log
I20260925 15:09:30.348330 12463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:30.327577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:33629 (local address 127.0.0.1:40167)
0925 15:09:30.328382 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:30.328394 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:30.328449 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:09:30.330004 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:30.330027 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:30.330476 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:30.330828 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:30.332632 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:30.333901 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:30.335633 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:30.336371 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:30.336517 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:30.341108 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:30.341116 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:30.341131 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:30.341171 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:30.344237 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:30.345033 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:30.345040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:30.345047 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:30.345208 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:30.345672 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:30.345676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:30.345678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:30.346198 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:30.346496 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:30.347025 (+ 529us) server_negotiation.cc:299] Negotiation successful
0925 15:09:30.347308 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":348,"threads_started":1}
I20260925 15:09:30.352221 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:30.354547 12460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b1157be78b4ba492f092c8c78a685f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } }
I20260925 15:09:30.354978 12460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:30.355087 12460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b1157be78b4ba492f092c8c78a685f, State: Initialized, Role: FOLLOWER
I20260925 15:09:30.355711 12460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b1157be78b4ba492f092c8c78a685f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } }
I20260925 15:09:30.355922 12460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:30.356047 12460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:30.356225 12460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:30.359153 12460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b1157be78b4ba492f092c8c78a685f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } }
I20260925 15:09:30.359762 12460 leader_election.cc:302] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67b1157be78b4ba492f092c8c78a685f; no voters:
I20260925 15:09:30.360426 12460 leader_election.cc:288] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:30.360587 12470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:30.362708 12460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:30.367295 12470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [term 1 LEADER]: Becoming Leader. State: Replica: 67b1157be78b4ba492f092c8c78a685f, State: Running, Role: LEADER
I20260925 15:09:30.368213 12470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b1157be78b4ba492f092c8c78a685f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } }
I20260925 15:09:30.371433 12471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67b1157be78b4ba492f092c8c78a685f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b1157be78b4ba492f092c8c78a685f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } } }
I20260925 15:09:30.371881 12471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:30.372469 12472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67b1157be78b4ba492f092c8c78a685f. Latest consensus state: current_term: 1 leader_uuid: "67b1157be78b4ba492f092c8c78a685f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b1157be78b4ba492f092c8c78a685f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } } }
I20260925 15:09:30.372771 12472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:30.374125 12463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:30.356510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:34947 (local address 127.0.0.1:40167)
0925 15:09:30.356694 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:30.356701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:30.357102 (+ 401us) server_negotiation.cc:407] Connection header received
0925 15:09:30.357265 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:30.357271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:30.357356 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:30.357459 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:30.358811 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:30.359904 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:30.361683 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:30.362310 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:30.362466 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:30.366082 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:30.366090 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:30.366095 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:30.366139 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:30.369847 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:30.370958 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:30.370963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:30.370967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:30.371108 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:30.371674 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:30.371679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:30.371682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:30.372103 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:30.372410 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:30.373544 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0925 15:09:30.373758 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:09:30.374773 12481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:30.379129 12481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:30.387789 12481 catalog_manager.cc:1347] Generated new cluster ID: aaba1fc3f2644ca2877e07471f6525c5
I20260925 15:09:30.387913 12481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:30.401096 12481 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:30.403000 12481 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:30.422240 12481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67b1157be78b4ba492f092c8c78a685f: Generated new TSK 0
I20260925 15:09:30.423571 12481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:30.435313 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12393
I20260925 15:09:30.436980 12455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:30.529327 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 12393
W20260925 15:09:30.543494 12378 connection.cc:535] client connection to 127.0.0.1:40167 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40167): Connection reset by peer (error 104)
2026-09-25T15:09:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1864 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:30 dist-test-slave-nsrc krb5kdc[12493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:30 dist-test-slave-nsrc krb5kdc[12493](info): set up 2 sockets
Sep 25 15:09:30 dist-test-slave-nsrc krb5kdc[12493](info): commencing operation
krb5kdc: starting...
W20260925 15:09:31.556530 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:31 dist-test-slave-nsrc krb5kdc[12493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:09:31.794890 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:32.111860 12506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:32.112178 12506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:32.112327 12506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:32.119534 12506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:32.119649 12506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:32.119727 12506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:32.119808 12506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:32.119868 12506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:32.129559 12506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:32.130766 12506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:32.132037 12506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:32.143514 12512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:32.143519 12514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:32.143600 12511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:32.144081 12506 server_base.cc:1034] running on GCE node
I20260925 15:09:32.144946 12506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:32.146239 12506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:32.147434 12506 hybrid_clock.cc:648] HybridClock initialized: now 1790348972147380 us; error 47 us; skew 500 ppm
I20260925 15:09:32.147758 12506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:32 dist-test-slave-nsrc krb5kdc[12493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:32.150669 12506 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:09:32.151974 12506 webserver.cc:460] Webserver started at http://127.0.0.1:37683/ using document root <none> and password file <none>
I20260925 15:09:32.152614 12506 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:32.152711 12506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:32.152943 12506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:32.155740 12506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "20cf2a92e53b44a3868de101cd92b625"
format_stamp: "Formatted at 2026-09-25 15:09:32 on dist-test-slave-nsrc"
server_key: "35031b803b6de0d338639fa3198ea02b"
server_key_iv: "a81e12ff6f633baaf4bc8225b27445c7"
server_key_version: "encryptionkey@0"
I20260925 15:09:32.156433 12506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "20cf2a92e53b44a3868de101cd92b625"
format_stamp: "Formatted at 2026-09-25 15:09:32 on dist-test-slave-nsrc"
server_key: "35031b803b6de0d338639fa3198ea02b"
server_key_iv: "a81e12ff6f633baaf4bc8225b27445c7"
server_key_version: "encryptionkey@0"
I20260925 15:09:32.161106 12506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:09:32.164258 12521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:32.165499 12506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:09:32.165624 12506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "20cf2a92e53b44a3868de101cd92b625"
format_stamp: "Formatted at 2026-09-25 15:09:32 on dist-test-slave-nsrc"
server_key: "35031b803b6de0d338639fa3198ea02b"
server_key_iv: "a81e12ff6f633baaf4bc8225b27445c7"
server_key_version: "encryptionkey@0"
I20260925 15:09:32.165773 12506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:32.180634 12506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:32.184121 12506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:32.184362 12506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:32.201288 12506 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41259
I20260925 15:09:32.201316 12572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41259 every 8 connection(s)
I20260925 15:09:32.202641 12506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:32.206223 12573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:32.206537 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 12506
I20260925 15:09:32.206686 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:32.207093 23472 external_mini_cluster.cc:1428] Setting key 1f2931aa1147caf91249b58933a48a01
I20260925 15:09:32.220057 12573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625: Bootstrap starting.
I20260925 15:09:32.221465 12576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:32.208795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:49113 (local address 127.0.0.1:41259)
0925 15:09:32.209581 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:32.209596 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:32.209644 (+ 48us) server_negotiation.cc:407] Connection header received
0925 15:09:32.212377 (+ 2733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:32.212406 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:32.212932 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:32.213375 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:32.213865 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:32.215628 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:32.216907 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:32.217930 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:32.218116 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:32.219132 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:32.219143 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:32.219160 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:32.219819 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:32.219925 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:32.219956 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:32.220115 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:32.220384 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 15:09:32.220536 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":302,"threads_started":1}
I20260925 15:09:32.223158 12573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:32.224277 12573 log.cc:824] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:32.225040 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:32.226830 12573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625: No bootstrap required, opened a new log
I20260925 15:09:32.235486 12576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:32.229183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:44913 (local address 127.0.0.1:41259)
0925 15:09:32.229366 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:32.229375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:32.229701 (+ 326us) server_negotiation.cc:407] Connection header received
0925 15:09:32.229842 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:32.229845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:32.229910 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:32.229985 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:32.230545 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:32.231282 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:32.233135 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:32.233583 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:32.233679 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:32.234825 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:32.234830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:32.234833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:32.234849 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:32.234884 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:32.234900 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:32.234977 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:32.235153 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 15:09:32.235192 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:09:32.235723 12573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20cf2a92e53b44a3868de101cd92b625" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } }
I20260925 15:09:32.236176 12573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:32.236284 12573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20cf2a92e53b44a3868de101cd92b625, State: Initialized, Role: FOLLOWER
I20260925 15:09:32.236912 12573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20cf2a92e53b44a3868de101cd92b625" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } }
I20260925 15:09:32.237147 12573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:32.237277 12573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:32.237416 12573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:32.239552 12573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20cf2a92e53b44a3868de101cd92b625" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } }
I20260925 15:09:32.240104 12573 leader_election.cc:302] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20cf2a92e53b44a3868de101cd92b625; no voters:
I20260925 15:09:32.240674 12573 leader_election.cc:288] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:32.240784 12583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:32.241427 12583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [term 1 LEADER]: Becoming Leader. State: Replica: 20cf2a92e53b44a3868de101cd92b625, State: Running, Role: LEADER
I20260925 15:09:32.241961 12583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20cf2a92e53b44a3868de101cd92b625" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } }
I20260925 15:09:32.242569 12573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:32.244565 12584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20cf2a92e53b44a3868de101cd92b625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20cf2a92e53b44a3868de101cd92b625" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } } }
I20260925 15:09:32.245069 12584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:32.244912 12585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20cf2a92e53b44a3868de101cd92b625. Latest consensus state: current_term: 1 leader_uuid: "20cf2a92e53b44a3868de101cd92b625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20cf2a92e53b44a3868de101cd92b625" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } } }
I20260925 15:09:32.245582 12585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:32.247403 12592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:32.252375 12592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:32.260749 12592 catalog_manager.cc:1347] Generated new cluster ID: b148a1e427db424f9677a0e67fdf29ab
I20260925 15:09:32.260860 12592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:32.284741 12592 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:32.285995 12592 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:32.294799 12592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20cf2a92e53b44a3868de101cd92b625: Generated new TSK 0
I20260925 15:09:32.295604 12592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:32.306869 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12506
W20260925 15:09:32.308013 12568 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.53:49113
W20260925 15:09:32.308756 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12506: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:09:32.308888 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 12506
W20260925 15:09:32.322189 12491 connection.cc:535] client connection to 127.0.0.1:41259 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41259): Connection reset by peer (error 104)
2026-09-25T15:09:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1778 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:32 dist-test-slave-nsrc krb5kdc[12604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:32 dist-test-slave-nsrc krb5kdc[12604](info): set up 2 sockets
Sep 25 15:09:32 dist-test-slave-nsrc krb5kdc[12604](info): commencing operation
krb5kdc: starting...
W20260925 15:09:33.320474 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:33 dist-test-slave-nsrc krb5kdc[12604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:09:33.557152 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:33.866405 12617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:33.866776 12617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:33.866922 12617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:33.874091 12617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:33.874209 12617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:33.874289 12617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:33.874369 12617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:33.874426 12617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:33.883965 12617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:33.885215 12617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:33.886430 12617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:33.897387 12622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:33.897434 12625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:33.897455 12623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:33.898501 12617 server_base.cc:1034] running on GCE node
I20260925 15:09:33.899214 12617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:33.900586 12617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:33.901778 12617 hybrid_clock.cc:648] HybridClock initialized: now 1790348973901728 us; error 38 us; skew 500 ppm
I20260925 15:09:33.902103 12617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:33 dist-test-slave-nsrc krb5kdc[12604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:33.910048 12617 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:09:33.911417 12617 webserver.cc:460] Webserver started at http://127.0.0.1:38759/ using document root <none> and password file <none>
I20260925 15:09:33.912083 12617 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:33.912220 12617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:33.912503 12617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:33.915320 12617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "df2899403eff42eebd17540b73ec329a"
format_stamp: "Formatted at 2026-09-25 15:09:33 on dist-test-slave-nsrc"
server_key: "1accf640e3a8453e1fb04ea0b3c43df2"
server_key_iv: "7a2f3d94af61f996a1b543b5d6fb2ba8"
server_key_version: "encryptionkey@0"
I20260925 15:09:33.916033 12617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "df2899403eff42eebd17540b73ec329a"
format_stamp: "Formatted at 2026-09-25 15:09:33 on dist-test-slave-nsrc"
server_key: "1accf640e3a8453e1fb04ea0b3c43df2"
server_key_iv: "7a2f3d94af61f996a1b543b5d6fb2ba8"
server_key_version: "encryptionkey@0"
I20260925 15:09:33.920917 12617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:09:33.924028 12632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:33.925264 12617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:09:33.925421 12617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "df2899403eff42eebd17540b73ec329a"
format_stamp: "Formatted at 2026-09-25 15:09:33 on dist-test-slave-nsrc"
server_key: "1accf640e3a8453e1fb04ea0b3c43df2"
server_key_iv: "7a2f3d94af61f996a1b543b5d6fb2ba8"
server_key_version: "encryptionkey@0"
I20260925 15:09:33.925587 12617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:33.930889 12617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:33.934791 12617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:33.935048 12617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:33.951577 12617 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46783
I20260925 15:09:33.951637 12683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46783 every 8 connection(s)
I20260925 15:09:33.952929 12617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:33.956523 12684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:33.957146 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 12617
I20260925 15:09:33.957298 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:33.957693 23472 external_mini_cluster.cc:1428] Setting key 30e6dc6ac9826f14359a648a99ee17d8
I20260925 15:09:33.966308 12687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:33.959412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:45695 (local address 127.0.0.1:46783)
0925 15:09:33.960226 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:33.960240 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:33.960290 (+ 50us) server_negotiation.cc:407] Connection header received
0925 15:09:33.962668 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:33.962695 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:33.963130 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:33.963568 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:33.964481 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:33.964489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:33.964505 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:33.964521 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:33.964580 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:33.964608 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:33.964822 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:33.964942 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 15:09:33.965165 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":280,"threads_started":1}
I20260925 15:09:33.969666 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:33.969834 12684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a: Bootstrap starting.
I20260925 15:09:33.973384 12684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:33.974570 12684 log.cc:824] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:33.976783 12687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:33.974550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:50383 (local address 127.0.0.1:46783)
0925 15:09:33.974779 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:33.974784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:33.975037 (+ 253us) server_negotiation.cc:407] Connection header received
0925 15:09:33.975231 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:33.975236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:33.975295 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:33.975403 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:33.976228 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:33.976231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:33.976234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:33.976248 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:33.976283 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:33.976296 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:33.976375 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:33.976501 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 15:09:33.976542 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:09:33.977592 12684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a: No bootstrap required, opened a new log
I20260925 15:09:33.984799 12684 raft_consensus.cc:348] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2899403eff42eebd17540b73ec329a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } }
I20260925 15:09:33.985131 12684 raft_consensus.cc:374] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:33.985224 12684 raft_consensus.cc:729] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2899403eff42eebd17540b73ec329a, State: Initialized, Role: FOLLOWER
I20260925 15:09:33.985735 12684 consensus_queue.cc:260] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2899403eff42eebd17540b73ec329a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } }
I20260925 15:09:33.985949 12684 raft_consensus.cc:388] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:33.986073 12684 raft_consensus.cc:482] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:33.986205 12684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:33.988274 12684 raft_consensus.cc:504] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2899403eff42eebd17540b73ec329a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } }
I20260925 15:09:33.988715 12684 leader_election.cc:302] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df2899403eff42eebd17540b73ec329a; no voters:
I20260925 15:09:33.989243 12684 leader_election.cc:288] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:33.989348 12694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:33.989966 12694 raft_consensus.cc:686] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [term 1 LEADER]: Becoming Leader. State: Replica: df2899403eff42eebd17540b73ec329a, State: Running, Role: LEADER
I20260925 15:09:33.990445 12694 consensus_queue.cc:237] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2899403eff42eebd17540b73ec329a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } }
I20260925 15:09:33.991315 12684 sys_catalog.cc:564] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:33.993162 12696 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [sys.catalog]: SysCatalogTable state changed. Reason: New leader df2899403eff42eebd17540b73ec329a. Latest consensus state: current_term: 1 leader_uuid: "df2899403eff42eebd17540b73ec329a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2899403eff42eebd17540b73ec329a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } } }
I20260925 15:09:33.993175 12695 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df2899403eff42eebd17540b73ec329a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2899403eff42eebd17540b73ec329a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } } }
I20260925 15:09:33.993585 12696 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:33.993650 12695 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:33.994364 12700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:33.999914 12700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:34.009245 12700 catalog_manager.cc:1347] Generated new cluster ID: 9774f06a9461449e8ccf569bebe14009
I20260925 15:09:34.009362 12700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:34.043596 12700 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:34.044003 12700 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:34.052676 12700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df2899403eff42eebd17540b73ec329a: Generated new TSK 0
I20260925 15:09:34.053669 12700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:34.066818 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12617
W20260925 15:09:34.067958 12679 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.53:45695
W20260925 15:09:34.068616 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12617: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:09:34.068764 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 12617
2026-09-25T15:09:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1759 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:34 dist-test-slave-nsrc krb5kdc[12715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:34 dist-test-slave-nsrc krb5kdc[12715](info): set up 2 sockets
Sep 25 15:09:34 dist-test-slave-nsrc krb5kdc[12715](info): commencing operation
krb5kdc: starting...
W20260925 15:09:35.080461 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:35 dist-test-slave-nsrc krb5kdc[12715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:09:35.308107 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:39833
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:32903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:39833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:35.622563 12728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:35.622836 12728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:35.622970 12728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:35.630030 12728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:35.630142 12728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:35.630220 12728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:35.630302 12728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:35.630360 12728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:35.639988 12728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:32903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:39833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:39833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:35.641249 12728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:35.642499 12728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:35.653410 12733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:35.653479 12736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:35.653642 12734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:35.654803 12728 server_base.cc:1034] running on GCE node
I20260925 15:09:35.655567 12728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:35.656996 12728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:35.658191 12728 hybrid_clock.cc:648] HybridClock initialized: now 1790348975658140 us; error 48 us; skew 500 ppm
I20260925 15:09:35.658537 12728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:35 dist-test-slave-nsrc krb5kdc[12715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348975, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:35.661486 12728 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:09:35.662914 12728 webserver.cc:460] Webserver started at http://127.22.236.62:42727/ using document root <none> and password file <none>
I20260925 15:09:35.663561 12728 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:35.663681 12728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:35.663952 12728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:35.666980 12728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "7de7c37ac9a14db6935d0fb2123098fa"
format_stamp: "Formatted at 2026-09-25 15:09:35 on dist-test-slave-nsrc"
server_key: "c0e969650ad8c4c7170445c890ebaeb8"
server_key_iv: "66d30e55eaa23a3574476b32ee079d59"
server_key_version: "encryptionkey@0"
I20260925 15:09:35.667706 12728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "7de7c37ac9a14db6935d0fb2123098fa"
format_stamp: "Formatted at 2026-09-25 15:09:35 on dist-test-slave-nsrc"
server_key: "c0e969650ad8c4c7170445c890ebaeb8"
server_key_iv: "66d30e55eaa23a3574476b32ee079d59"
server_key_version: "encryptionkey@0"
I20260925 15:09:35.672570 12728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 15:09:35.675748 12743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:35.677029 12728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:35.677191 12728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "7de7c37ac9a14db6935d0fb2123098fa"
format_stamp: "Formatted at 2026-09-25 15:09:35 on dist-test-slave-nsrc"
server_key: "c0e969650ad8c4c7170445c890ebaeb8"
server_key_iv: "66d30e55eaa23a3574476b32ee079d59"
server_key_version: "encryptionkey@0"
I20260925 15:09:35.677358 12728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:35.694765 12728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:35.699036 12728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:35.699291 12728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:35.715941 12728 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:39833
I20260925 15:09:35.715961 12794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:39833 every 8 connection(s)
I20260925 15:09:35.717214 12728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:35.718124 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 12728
I20260925 15:09:35.718302 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:35.718838 23472 external_mini_cluster.cc:1428] Setting key eac3434f20f2eeed3d2e6fe2bac18492
I20260925 15:09:35.721522 12795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:09:35 dist-test-slave-nsrc krb5kdc[12715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:09:35.731700 12795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa: Bootstrap starting.
I20260925 15:09:35.735009 12795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:35.736050 12795 log.cc:824] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:35.739154 12795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa: No bootstrap required, opened a new log
I20260925 15:09:35.739899 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:35.720643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57801 (local address 127.22.236.62:39833)
0925 15:09:35.721352 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:35.721364 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:35.721410 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:09:35.722997 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:35.723018 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:35.723380 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:35.723742 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:35.725162 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:35.726497 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:35.727751 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:35.728438 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:35.728603 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:35.732872 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:35.732880 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:35.732896 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:35.732930 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:35.735815 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:35.736778 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:35.736787 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:35.736795 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:35.737013 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:35.737426 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:35.737430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:35.737432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:35.737903 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:35.738118 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:35.738598 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 15:09:35.738864 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":323,"threads_started":1}
I20260925 15:09:35.743487 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:35.748163 12795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de7c37ac9a14db6935d0fb2123098fa" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39833 } }
I20260925 15:09:35.748524 12795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:35.748617 12795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de7c37ac9a14db6935d0fb2123098fa, State: Initialized, Role: FOLLOWER
I20260925 15:09:35.749327 12795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de7c37ac9a14db6935d0fb2123098fa" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39833 } }
I20260925 15:09:35.749604 12795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:35.749747 12795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:35.749922 12795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:35.752738 12795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de7c37ac9a14db6935d0fb2123098fa" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39833 } }
I20260925 15:09:35.753352 12795 leader_election.cc:302] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7de7c37ac9a14db6935d0fb2123098fa; no voters:
I20260925 15:09:35.754036 12795 leader_election.cc:288] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:35.754143 12805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:35.754783 12805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [term 1 LEADER]: Becoming Leader. State: Replica: 7de7c37ac9a14db6935d0fb2123098fa, State: Running, Role: LEADER
I20260925 15:09:35.755378 12805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de7c37ac9a14db6935d0fb2123098fa" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39833 } }
I20260925 15:09:35.756561 12795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:35.763481 12806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7de7c37ac9a14db6935d0fb2123098fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de7c37ac9a14db6935d0fb2123098fa" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39833 } } }
I20260925 15:09:35.763976 12806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:35.763896 12807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7de7c37ac9a14db6935d0fb2123098fa. Latest consensus state: current_term: 1 leader_uuid: "7de7c37ac9a14db6935d0fb2123098fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de7c37ac9a14db6935d0fb2123098fa" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 39833 } } }
I20260925 15:09:35.764252 12807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:35.769279 12810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:35.770498 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:35.748263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52827 (local address 127.22.236.62:39833)
0925 15:09:35.748414 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:35.748420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:35.749156 (+ 736us) server_negotiation.cc:407] Connection header received
0925 15:09:35.749322 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:35.749325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:35.749382 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:35.749456 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:35.751263 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:35.752024 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:35.753698 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:35.754179 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:35.754292 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:35.757223 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:35.757230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:35.757235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:35.757276 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:35.766998 (+ 9722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:35.767979 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:35.767985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:35.767988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:35.768231 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:35.768808 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:35.768813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:35.768816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:35.769217 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:35.769561 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:35.769893 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 15:09:35.770112 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:09:35.774268 12810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:35.782735 12810 catalog_manager.cc:1347] Generated new cluster ID: b3ec21228212476da659e8023df18ac6
I20260925 15:09:35.782852 12810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:35.829035 12810 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:35.830945 12810 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:35.840602 12810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7de7c37ac9a14db6935d0fb2123098fa: Generated new TSK 0
I20260925 15:09:35.841563 12810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:35.858883 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12728
I20260925 15:09:35.860529 12790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:35.953684 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 12728
W20260925 15:09:35.967406 12713 connection.cc:535] client connection to 127.22.236.62:39833 recv error: Network error: failed to read from TLS socket (remote: 127.22.236.62:39833): Connection reset by peer (error 104)
2026-09-25T15:09:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1885 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:36 dist-test-slave-nsrc krb5kdc[12827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:36 dist-test-slave-nsrc krb5kdc[12827](info): set up 2 sockets
Sep 25 15:09:36 dist-test-slave-nsrc krb5kdc[12827](info): commencing operation
krb5kdc: starting...
W20260925 15:09:36.952505 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:37 dist-test-slave-nsrc krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:09:37.179370 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:37061
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:45125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:37061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:37.494207 12840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:37.494495 12840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:37.494624 12840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:37.501811 12840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:37.501924 12840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:37.502002 12840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:37.502089 12840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:37.502141 12840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:37.511950 12840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:45125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:37061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:37061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:37.513202 12840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:37.514456 12840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:37.525439 12846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:37.525490 12848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:37.525514 12845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:37.526361 12840 server_base.cc:1034] running on GCE node
I20260925 15:09:37.527109 12840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:37.528486 12840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:37.529663 12840 hybrid_clock.cc:648] HybridClock initialized: now 1790348977529617 us; error 43 us; skew 500 ppm
I20260925 15:09:37.529983 12840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:37 dist-test-slave-nsrc krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348977, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:37.533020 12840 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:09:37.534353 12840 webserver.cc:460] Webserver started at http://127.22.236.62:41155/ using document root <none> and password file <none>
I20260925 15:09:37.535121 12840 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:37.535267 12840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:37.535596 12840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:37.539153 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "253207d2a385437fa6e2764566d81e7a"
format_stamp: "Formatted at 2026-09-25 15:09:37 on dist-test-slave-nsrc"
server_key: "ed67a19a670d5da7a53a4ac6db4af06f"
server_key_iv: "978c3c58fd80ae87e45704c242bc2383"
server_key_version: "encryptionkey@0"
I20260925 15:09:37.539866 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "253207d2a385437fa6e2764566d81e7a"
format_stamp: "Formatted at 2026-09-25 15:09:37 on dist-test-slave-nsrc"
server_key: "ed67a19a670d5da7a53a4ac6db4af06f"
server_key_iv: "978c3c58fd80ae87e45704c242bc2383"
server_key_version: "encryptionkey@0"
I20260925 15:09:37.545040 12840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:09:37.548370 12855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:37.549656 12840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:09:37.549803 12840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "253207d2a385437fa6e2764566d81e7a"
format_stamp: "Formatted at 2026-09-25 15:09:37 on dist-test-slave-nsrc"
server_key: "ed67a19a670d5da7a53a4ac6db4af06f"
server_key_iv: "978c3c58fd80ae87e45704c242bc2383"
server_key_version: "encryptionkey@0"
I20260925 15:09:37.549963 12840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:37.564890 12840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:37.568511 12840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:37.568728 12840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:37.584797 12840 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:37061
I20260925 15:09:37.584827 12906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:37061 every 8 connection(s)
I20260925 15:09:37.586040 12840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:37.589603 12907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:37.591943 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 12840
I20260925 15:09:37.592094 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:37.592504 23472 external_mini_cluster.cc:1428] Setting key c74d8bb04d27778d8f1060ecf160da45
I20260925 15:09:37.602103 12907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a: Bootstrap starting.
I20260925 15:09:37.605881 12907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:37.607082 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:37.594289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60919 (local address 127.22.236.62:37061)
0925 15:09:37.595067 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:37.595082 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:37.595133 (+ 51us) server_negotiation.cc:407] Connection header received
0925 15:09:37.597390 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:37.597416 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:37.597902 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:37.598348 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:37.598852 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:37.600891 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:37.602217 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:37.603182 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:37.603360 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:37.604433 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:37.604443 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:37.604461 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:37.605142 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:37.605237 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:37.605273 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:37.605418 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:37.605628 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 15:09:37.605793 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":311,"threads_started":1}
I20260925 15:09:37.607167 12907 log.cc:824] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:37.610311 12907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a: No bootstrap required, opened a new log
I20260925 15:09:37.611312 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:37.618487 12907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253207d2a385437fa6e2764566d81e7a" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37061 } }
I20260925 15:09:37.618842 12907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:37.618945 12907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253207d2a385437fa6e2764566d81e7a, State: Initialized, Role: FOLLOWER
I20260925 15:09:37.619580 12907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253207d2a385437fa6e2764566d81e7a" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37061 } }
I20260925 15:09:37.619863 12907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:37.620013 12907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:37.620249 12907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:37.621865 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:37.615436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47053 (local address 127.22.236.62:37061)
0925 15:09:37.615601 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:37.615606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:37.615923 (+ 317us) server_negotiation.cc:407] Connection header received
0925 15:09:37.616075 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:37.616081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:37.616173 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:37.616277 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:37.616684 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:37.617821 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:37.619231 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:37.619869 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:37.620014 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:37.621096 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:37.621103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:37.621108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:37.621129 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:37.621193 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:37.621214 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:37.621327 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:37.621495 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 15:09:37.621553 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:09:37.623078 12907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253207d2a385437fa6e2764566d81e7a" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37061 } }
I20260925 15:09:37.623612 12907 leader_election.cc:302] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 253207d2a385437fa6e2764566d81e7a; no voters:
I20260925 15:09:37.624130 12907 leader_election.cc:288] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:37.624280 12917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:37.624847 12917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [term 1 LEADER]: Becoming Leader. State: Replica: 253207d2a385437fa6e2764566d81e7a, State: Running, Role: LEADER
I20260925 15:09:37.625396 12917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253207d2a385437fa6e2764566d81e7a" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37061 } }
I20260925 15:09:37.627017 12917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 253207d2a385437fa6e2764566d81e7a. Latest consensus state: current_term: 1 leader_uuid: "253207d2a385437fa6e2764566d81e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253207d2a385437fa6e2764566d81e7a" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37061 } } }
I20260925 15:09:37.627269 12917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:37.627727 12919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "253207d2a385437fa6e2764566d81e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253207d2a385437fa6e2764566d81e7a" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 37061 } } }
I20260925 15:09:37.628036 12919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:37.629683 12907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:37.633281 12921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:37.638823 12921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:37.648051 12921 catalog_manager.cc:1347] Generated new cluster ID: 0ebfde2f3d5949678c9d820c5409dba5
I20260925 15:09:37.648185 12921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:37.658785 12921 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:37.660636 12921 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:37.666986 12921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 253207d2a385437fa6e2764566d81e7a: Generated new TSK 0
I20260925 15:09:37.667972 12921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:37.680096 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12840
W20260925 15:09:37.681262 12902 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60919
W20260925 15:09:37.681967 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12840: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:09:37.682123 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 12840
W20260925 15:09:37.695524 12825 connection.cc:535] client connection to 127.22.236.62:37061 recv error: Network error: failed to read from TLS socket (remote: 127.22.236.62:37061): Connection reset by peer (error 104)
2026-09-25T15:09:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1728 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:37 dist-test-slave-nsrc krb5kdc[12938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:37 dist-test-slave-nsrc krb5kdc[12938](info): set up 2 sockets
Sep 25 15:09:37 dist-test-slave-nsrc krb5kdc[12938](info): commencing operation
krb5kdc: starting...
W20260925 15:09:38.684401 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:38 dist-test-slave-nsrc krb5kdc[12938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:09:38.907714 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:35261
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:35261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:39.219659 12951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:39.219935 12951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:39.220067 12951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:39.227027 12951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:39.227140 12951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:39.227216 12951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:39.227301 12951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:39.227358 12951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:39.236789 12951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:35261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:35261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:39.237953 12951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:39.239177 12951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:39.249715 12957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:39.249935 12956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:39.249752 12959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:39.250758 12951 server_base.cc:1034] running on GCE node
I20260925 15:09:39.251422 12951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:39.252686 12951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:39.253897 12951 hybrid_clock.cc:648] HybridClock initialized: now 1790348979253852 us; error 36 us; skew 500 ppm
I20260925 15:09:39.254210 12951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:39 dist-test-slave-nsrc krb5kdc[12938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348979, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:39.261708 12951 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:09:39.263216 12951 webserver.cc:460] Webserver started at http://127.22.236.62:35969/ using document root <none> and password file <none>
I20260925 15:09:39.263881 12951 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:39.263998 12951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:39.264330 12951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:39.267243 12951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "823d1dd6ad7f439f8a0fbd46b52957b8"
format_stamp: "Formatted at 2026-09-25 15:09:39 on dist-test-slave-nsrc"
server_key: "9e9c8cfb776694573ffcc2349923b923"
server_key_iv: "076a1c76782a47b4182d02b8e7054b59"
server_key_version: "encryptionkey@0"
I20260925 15:09:39.268008 12951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "823d1dd6ad7f439f8a0fbd46b52957b8"
format_stamp: "Formatted at 2026-09-25 15:09:39 on dist-test-slave-nsrc"
server_key: "9e9c8cfb776694573ffcc2349923b923"
server_key_iv: "076a1c76782a47b4182d02b8e7054b59"
server_key_version: "encryptionkey@0"
I20260925 15:09:39.273149 12951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:09:39.276463 12966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:39.277701 12951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:09:39.277877 12951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "823d1dd6ad7f439f8a0fbd46b52957b8"
format_stamp: "Formatted at 2026-09-25 15:09:39 on dist-test-slave-nsrc"
server_key: "9e9c8cfb776694573ffcc2349923b923"
server_key_iv: "076a1c76782a47b4182d02b8e7054b59"
server_key_version: "encryptionkey@0"
I20260925 15:09:39.278050 12951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:39.283424 12951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:39.287109 12951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:39.287361 12951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:39.304613 12951 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:35261
I20260925 15:09:39.304657 13017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:35261 every 8 connection(s)
I20260925 15:09:39.305955 12951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:39.307426 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 12951
I20260925 15:09:39.307581 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:39.308007 23472 external_mini_cluster.cc:1428] Setting key b4b6a6d15d4cbe7d15d6e81eb3099309
I20260925 15:09:39.309782 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:39.315338 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:39.309694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53303 (local address 127.22.236.62:35261)
0925 15:09:39.310441 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:39.310452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:39.310492 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:09:39.312080 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:39.312100 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:39.312512 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:39.312893 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:39.313844 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:39.313850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:39.313862 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:39.313876 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:39.313950 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:39.313969 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:39.314173 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:39.314319 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 15:09:39.314483 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":319,"threads_started":1}
I20260925 15:09:39.318548 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:39.320654 13018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8: Bootstrap starting.
I20260925 15:09:39.323891 13018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:39.324782 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:39.322577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39389 (local address 127.22.236.62:35261)
0925 15:09:39.322728 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:39.322734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:39.323060 (+ 326us) server_negotiation.cc:407] Connection header received
0925 15:09:39.323200 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:39.323205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:39.323251 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:39.323327 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:39.324176 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:39.324182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:39.324186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:39.324204 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:39.324247 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:39.324265 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:39.324363 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:39.324472 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 15:09:39.324524 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:09:39.325069 13018 log.cc:824] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:39.327579 13018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8: No bootstrap required, opened a new log
I20260925 15:09:39.334635 13018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823d1dd6ad7f439f8a0fbd46b52957b8" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35261 } }
I20260925 15:09:39.334954 13018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:39.335041 13018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823d1dd6ad7f439f8a0fbd46b52957b8, State: Initialized, Role: FOLLOWER
I20260925 15:09:39.335563 13018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823d1dd6ad7f439f8a0fbd46b52957b8" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35261 } }
I20260925 15:09:39.335776 13018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:39.335892 13018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:39.336036 13018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:39.338323 13018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823d1dd6ad7f439f8a0fbd46b52957b8" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35261 } }
I20260925 15:09:39.338773 13018 leader_election.cc:302] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 823d1dd6ad7f439f8a0fbd46b52957b8; no voters:
I20260925 15:09:39.339264 13018 leader_election.cc:288] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:39.339372 13028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:39.339975 13028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [term 1 LEADER]: Becoming Leader. State: Replica: 823d1dd6ad7f439f8a0fbd46b52957b8, State: Running, Role: LEADER
I20260925 15:09:39.340494 13028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823d1dd6ad7f439f8a0fbd46b52957b8" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35261 } }
I20260925 15:09:39.341117 13018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:39.343029 13029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "823d1dd6ad7f439f8a0fbd46b52957b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823d1dd6ad7f439f8a0fbd46b52957b8" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35261 } } }
I20260925 15:09:39.343483 13029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:39.343359 13030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 823d1dd6ad7f439f8a0fbd46b52957b8. Latest consensus state: current_term: 1 leader_uuid: "823d1dd6ad7f439f8a0fbd46b52957b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823d1dd6ad7f439f8a0fbd46b52957b8" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 35261 } } }
I20260925 15:09:39.343607 13030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:39.345091 13036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:39.349977 13036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:39.359058 13036 catalog_manager.cc:1347] Generated new cluster ID: 582c2654e3d041fc97832df7b35e097f
I20260925 15:09:39.359154 13036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:39.386488 13036 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:39.387046 13036 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:39.394893 13036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 823d1dd6ad7f439f8a0fbd46b52957b8: Generated new TSK 0
I20260925 15:09:39.395965 13036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:39.415055 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12951
W20260925 15:09:39.416335 13013 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53303
W20260925 15:09:39.417029 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 12951: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:09:39.417160 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 12951
2026-09-25T15:09:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1735 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:39 dist-test-slave-nsrc krb5kdc[13049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:39 dist-test-slave-nsrc krb5kdc[13049](info): set up 2 sockets
Sep 25 15:09:39 dist-test-slave-nsrc krb5kdc[13049](info): commencing operation
krb5kdc: starting...
W20260925 15:09:40.428478 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:40 dist-test-slave-nsrc krb5kdc[13049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:09:40.657367 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:45185
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:45185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:40.966809 13062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:40.967082 13062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:40.967212 13062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:40.974329 13062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:40.974447 13062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:40.974527 13062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:40.974603 13062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:40.974661 13062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:40.984313 13062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:45185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:45185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:40.985634 13062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:40.986855 13062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:40.997457 13067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:40.997526 13070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:40.997474 13068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:40.998615 13062 server_base.cc:1034] running on GCE node
I20260925 15:09:40.999380 13062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:41.000794 13062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:41.001988 13062 hybrid_clock.cc:648] HybridClock initialized: now 1790348981001939 us; error 46 us; skew 500 ppm
I20260925 15:09:41.002303 13062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:41 dist-test-slave-nsrc krb5kdc[13049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348981, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:41.005241 13062 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:09:41.006680 13062 webserver.cc:460] Webserver started at http://127.22.236.62:40291/ using document root <none> and password file <none>
I20260925 15:09:41.007289 13062 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:41.007385 13062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:41.007623 13062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:41.010447 13062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "20da976763014ec6855125cd791d69c6"
format_stamp: "Formatted at 2026-09-25 15:09:41 on dist-test-slave-nsrc"
server_key: "3c2b8c505b4d242911718114e3f5c85f"
server_key_iv: "e017682acb34eac9e4913f0585ae1a32"
server_key_version: "encryptionkey@0"
I20260925 15:09:41.011111 13062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "20da976763014ec6855125cd791d69c6"
format_stamp: "Formatted at 2026-09-25 15:09:41 on dist-test-slave-nsrc"
server_key: "3c2b8c505b4d242911718114e3f5c85f"
server_key_iv: "e017682acb34eac9e4913f0585ae1a32"
server_key_version: "encryptionkey@0"
I20260925 15:09:41.015888 13062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260925 15:09:41.019004 13077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:41.020279 13062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:41.020421 13062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "20da976763014ec6855125cd791d69c6"
format_stamp: "Formatted at 2026-09-25 15:09:41 on dist-test-slave-nsrc"
server_key: "3c2b8c505b4d242911718114e3f5c85f"
server_key_iv: "e017682acb34eac9e4913f0585ae1a32"
server_key_version: "encryptionkey@0"
I20260925 15:09:41.020567 13062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:41.043499 13062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:41.047138 13062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:41.047358 13062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:41.063584 13062 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:45185
I20260925 15:09:41.063607 13128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:45185 every 8 connection(s)
I20260925 15:09:41.064914 13062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:41.067426 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 13062
I20260925 15:09:41.067592 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:41.068163 23472 external_mini_cluster.cc:1428] Setting key 1601a67a71670e033b5bab3ec9dfe275
I20260925 15:09:41.068934 13129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:09:41 dist-test-slave-nsrc krb5kdc[13049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:09:41.083128 13129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6: Bootstrap starting.
I20260925 15:09:41.087311 13129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:41.088668 13129 log.cc:824] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:41.092382 13129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6: No bootstrap required, opened a new log
I20260925 15:09:41.092751 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:41.069869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47145 (local address 127.22.236.62:45185)
0925 15:09:41.070596 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:41.070607 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:41.070647 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:09:41.072206 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:41.072229 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:41.072651 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:41.072999 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:41.074721 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:41.076224 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:41.077928 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:41.078620 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:41.078745 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:41.083474 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:41.083482 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:41.083497 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:41.083534 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:41.087252 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:41.088171 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:41.088181 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:41.088189 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:41.088418 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:41.088863 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:41.088868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:41.088872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:41.089622 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:41.089906 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:41.091004 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0925 15:09:41.091428 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":311,"threads_started":1}
I20260925 15:09:41.096321 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:41.099870 13129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20da976763014ec6855125cd791d69c6" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45185 } }
I20260925 15:09:41.100327 13129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:41.100432 13129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20da976763014ec6855125cd791d69c6, State: Initialized, Role: FOLLOWER
I20260925 15:09:41.101054 13129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20da976763014ec6855125cd791d69c6" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45185 } }
I20260925 15:09:41.101332 13129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:41.101468 13129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:41.101619 13129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:41.103680 13129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20da976763014ec6855125cd791d69c6" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45185 } }
I20260925 15:09:41.104156 13129 leader_election.cc:302] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20da976763014ec6855125cd791d69c6; no voters:
I20260925 15:09:41.104720 13129 leader_election.cc:288] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:41.104856 13139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:41.105849 13139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [term 1 LEADER]: Becoming Leader. State: Replica: 20da976763014ec6855125cd791d69c6, State: Running, Role: LEADER
I20260925 15:09:41.106323 13139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20da976763014ec6855125cd791d69c6" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45185 } }
I20260925 15:09:41.106894 13129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:41.108805 13141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20da976763014ec6855125cd791d69c6. Latest consensus state: current_term: 1 leader_uuid: "20da976763014ec6855125cd791d69c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20da976763014ec6855125cd791d69c6" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45185 } } }
I20260925 15:09:41.109228 13141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:41.109588 13140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20da976763014ec6855125cd791d69c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20da976763014ec6855125cd791d69c6" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 45185 } } }
I20260925 15:09:41.110128 13140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:41.116300 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:41.100331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39265 (local address 127.22.236.62:45185)
0925 15:09:41.100508 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:41.100514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:41.101026 (+ 512us) server_negotiation.cc:407] Connection header received
0925 15:09:41.101187 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:41.101190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:41.101248 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:41.101332 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:41.102752 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:41.103460 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:41.104956 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:41.105672 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:41.105797 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:41.108843 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:41.108848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:41.108853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:41.108896 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:41.112401 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:41.113236 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:41.113241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:41.113245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:41.113398 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:41.113865 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:41.113875 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:41.113879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:41.114273 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:41.115491 (+ 1218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:41.115640 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 15:09:41.115866 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:09:41.119604 13148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:41.125123 13148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:41.134109 13148 catalog_manager.cc:1347] Generated new cluster ID: 3db435248e8f4c58bad938537d2c4b04
I20260925 15:09:41.134219 13148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:41.161864 13148 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:41.163167 13148 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:41.176623 13148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20da976763014ec6855125cd791d69c6: Generated new TSK 0
I20260925 15:09:41.177654 13148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:41.191745 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 13062
I20260925 15:09:41.193707 13124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:41.285001 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 13062
2026-09-25T15:09:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1867 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:41 dist-test-slave-nsrc krb5kdc[13161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:41 dist-test-slave-nsrc krb5kdc[13161](info): set up 2 sockets
Sep 25 15:09:41 dist-test-slave-nsrc krb5kdc[13161](info): commencing operation
krb5kdc: starting...
W20260925 15:09:42.296491 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:42 dist-test-slave-nsrc krb5kdc[13161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:09:42.525528 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:36241
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:36241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:42.839761 13174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:42.840062 13174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:42.840220 13174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:42.847227 13174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:42.847339 13174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:42.847419 13174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:42.847499 13174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:42.847559 13174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:42.857013 13174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:36241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:36241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:42.858210 13174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:42.859467 13174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:42.870383 13179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:42.870426 13180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:42.870451 13182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:42.871169 13174 server_base.cc:1034] running on GCE node
I20260925 15:09:42.871876 13174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:42.873313 13174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:42.874516 13174 hybrid_clock.cc:648] HybridClock initialized: now 1790348982874467 us; error 46 us; skew 500 ppm
I20260925 15:09:42.874827 13174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:42 dist-test-slave-nsrc krb5kdc[13161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348982, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:42.877718 13174 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:09:42.879048 13174 webserver.cc:460] Webserver started at http://127.22.236.62:32809/ using document root <none> and password file <none>
I20260925 15:09:42.879657 13174 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:42.879755 13174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:42.880000 13174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:42.882840 13174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "a3c3f2db6b0d4d5d8e36689dfa9ba90b"
format_stamp: "Formatted at 2026-09-25 15:09:42 on dist-test-slave-nsrc"
server_key: "013857d5c1c9e39e06dc465b527bd2e7"
server_key_iv: "c0da45da984bec2e62dc61b65a55bf62"
server_key_version: "encryptionkey@0"
I20260925 15:09:42.883546 13174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "a3c3f2db6b0d4d5d8e36689dfa9ba90b"
format_stamp: "Formatted at 2026-09-25 15:09:42 on dist-test-slave-nsrc"
server_key: "013857d5c1c9e39e06dc465b527bd2e7"
server_key_iv: "c0da45da984bec2e62dc61b65a55bf62"
server_key_version: "encryptionkey@0"
I20260925 15:09:42.888391 13174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:09:42.891541 13189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:42.892788 13174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:42.892948 13174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "a3c3f2db6b0d4d5d8e36689dfa9ba90b"
format_stamp: "Formatted at 2026-09-25 15:09:42 on dist-test-slave-nsrc"
server_key: "013857d5c1c9e39e06dc465b527bd2e7"
server_key_iv: "c0da45da984bec2e62dc61b65a55bf62"
server_key_version: "encryptionkey@0"
I20260925 15:09:42.893110 13174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:42.918418 13174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:42.922243 13174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:42.922500 13174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:42.938880 13174 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:36241
I20260925 15:09:42.938905 13240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:36241 every 8 connection(s)
I20260925 15:09:42.940214 13174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:42.943715 13241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:42.947230 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 13174
I20260925 15:09:42.947402 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:42.947924 23472 external_mini_cluster.cc:1428] Setting key 2b127dffebe3c9b42cf66c717851f8cd
I20260925 15:09:42.955643 13241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b: Bootstrap starting.
I20260925 15:09:42.959065 13241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:42.960206 13241 log.cc:824] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:42.960588 13244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:42.949666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50613 (local address 127.22.236.62:36241)
0925 15:09:42.950359 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:42.950370 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:42.950411 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:09:42.951998 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:42.952020 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:42.952467 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:42.952778 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:42.953316 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:42.954616 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:42.956386 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:42.957041 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:42.957177 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:42.958359 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:42.958366 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:42.958379 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:42.958865 (+ 486us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:42.958946 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:42.958973 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:42.959055 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:42.959301 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 15:09:42.959556 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":314,"threads_started":1}
I20260925 15:09:42.962908 13241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b: No bootstrap required, opened a new log
I20260925 15:09:42.964068 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:42.972564 13241 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c3f2db6b0d4d5d8e36689dfa9ba90b" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36241 } }
I20260925 15:09:42.973027 13241 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:42.973130 13241 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c3f2db6b0d4d5d8e36689dfa9ba90b, State: Initialized, Role: FOLLOWER
I20260925 15:09:42.973685 13241 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c3f2db6b0d4d5d8e36689dfa9ba90b" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36241 } }
I20260925 15:09:42.973904 13241 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:42.974032 13241 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:42.974184 13241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:42.975126 13244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:42.969006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42277 (local address 127.22.236.62:36241)
0925 15:09:42.969187 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:42.969192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:42.969508 (+ 316us) server_negotiation.cc:407] Connection header received
0925 15:09:42.969673 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:42.969677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:42.969740 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:42.969822 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:42.970333 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:42.971058 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:42.972844 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:42.973327 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:42.973441 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:42.974449 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:42.974455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:42.974460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:42.974479 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:42.974516 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:42.974530 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:42.974618 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:42.974748 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 15:09:42.974831 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:09:42.976686 13241 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c3f2db6b0d4d5d8e36689dfa9ba90b" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36241 } }
I20260925 15:09:42.977157 13241 leader_election.cc:302] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3c3f2db6b0d4d5d8e36689dfa9ba90b; no voters:
I20260925 15:09:42.977676 13241 leader_election.cc:288] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:42.977736 13253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:42.979581 13241 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:42.980448 13253 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [term 1 LEADER]: Becoming Leader. State: Replica: a3c3f2db6b0d4d5d8e36689dfa9ba90b, State: Running, Role: LEADER
I20260925 15:09:42.981014 13253 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c3f2db6b0d4d5d8e36689dfa9ba90b" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36241 } }
I20260925 15:09:42.983848 13254 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3c3f2db6b0d4d5d8e36689dfa9ba90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c3f2db6b0d4d5d8e36689dfa9ba90b" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36241 } } }
I20260925 15:09:42.984292 13254 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:42.983824 13255 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3c3f2db6b0d4d5d8e36689dfa9ba90b. Latest consensus state: current_term: 1 leader_uuid: "a3c3f2db6b0d4d5d8e36689dfa9ba90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c3f2db6b0d4d5d8e36689dfa9ba90b" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36241 } } }
I20260925 15:09:42.984620 13255 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:42.985814 13263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:42.990473 13263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:42.999782 13263 catalog_manager.cc:1347] Generated new cluster ID: 085e7f520aa840d68f2cff6dd9c5ac61
I20260925 15:09:42.999907 13263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:43.012985 13263 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:43.014253 13263 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:43.023398 13263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3c3f2db6b0d4d5d8e36689dfa9ba90b: Generated new TSK 0
I20260925 15:09:43.024466 13263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:43.041076 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 13174
W20260925 15:09:43.042109 13236 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50613
W20260925 15:09:43.042796 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 13174: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:09:43.042941 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 13174
2026-09-25T15:09:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1757 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:43 dist-test-slave-nsrc krb5kdc[13274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:43 dist-test-slave-nsrc krb5kdc[13274](info): set up 2 sockets
Sep 25 15:09:43 dist-test-slave-nsrc krb5kdc[13274](info): commencing operation
krb5kdc: starting...
W20260925 15:09:44.052480 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.007s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:44 dist-test-slave-nsrc krb5kdc[13274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:09:44.275406 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:38553
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:38553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:44.591646 13287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:44.591936 13287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:44.592077 13287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:44.599155 13287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:44.599277 13287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:44.599355 13287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:44.599445 13287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:44.599504 13287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:44.609138 13287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:38553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:38553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:44.610369 13287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:44.611634 13287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:44.622287 13292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:44.622241 13295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:44.622310 13293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:44.623386 13287 server_base.cc:1034] running on GCE node
I20260925 15:09:44.624085 13287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:44.625432 13287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:44.626632 13287 hybrid_clock.cc:648] HybridClock initialized: now 1790348984626587 us; error 33 us; skew 500 ppm
I20260925 15:09:44.626953 13287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:44 dist-test-slave-nsrc krb5kdc[13274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348984, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:44.633437 13287 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:09:44.634769 13287 webserver.cc:460] Webserver started at http://127.22.236.62:43161/ using document root <none> and password file <none>
I20260925 15:09:44.635391 13287 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:44.635504 13287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:44.635784 13287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:44.638648 13287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "f178a398df1142308cc15390d1e15aad"
format_stamp: "Formatted at 2026-09-25 15:09:44 on dist-test-slave-nsrc"
server_key: "700c38a319e5719a78006e69aa250899"
server_key_iv: "c2f6d39eadfec95873c2efa1c8a78348"
server_key_version: "encryptionkey@0"
I20260925 15:09:44.639362 13287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "f178a398df1142308cc15390d1e15aad"
format_stamp: "Formatted at 2026-09-25 15:09:44 on dist-test-slave-nsrc"
server_key: "700c38a319e5719a78006e69aa250899"
server_key_iv: "c2f6d39eadfec95873c2efa1c8a78348"
server_key_version: "encryptionkey@0"
I20260925 15:09:44.644235 13287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260925 15:09:44.647452 13302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:44.648766 13287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 15:09:44.648929 13287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "f178a398df1142308cc15390d1e15aad"
format_stamp: "Formatted at 2026-09-25 15:09:44 on dist-test-slave-nsrc"
server_key: "700c38a319e5719a78006e69aa250899"
server_key_iv: "c2f6d39eadfec95873c2efa1c8a78348"
server_key_version: "encryptionkey@0"
I20260925 15:09:44.649099 13287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:44.654388 13287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:44.658043 13287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:44.658293 13287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:44.674674 13287 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:38553
I20260925 15:09:44.674713 13354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:38553 every 8 connection(s)
I20260925 15:09:44.675937 13287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:44.679361 13355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:44.679911 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 13287
I20260925 15:09:44.680096 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:44.680640 23472 external_mini_cluster.cc:1428] Setting key 5a26128933cf5bb0522a4443800f22b3
I20260925 15:09:44.688683 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:44.682392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45455 (local address 127.22.236.62:38553)
0925 15:09:44.683164 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:44.683175 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:44.683215 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:09:44.685437 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:44.685459 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:44.685805 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:44.686151 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:44.687131 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:44.687137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:44.687149 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:44.687164 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:44.687222 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:44.687241 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:44.687447 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:44.687571 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 15:09:44.687749 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":346,"threads_started":1}
I20260925 15:09:44.691835 13355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad: Bootstrap starting.
I20260925 15:09:44.692167 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:44.694928 13355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:44.695941 13355 log.cc:824] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:44.698577 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:44.696261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52667 (local address 127.22.236.62:38553)
0925 15:09:44.696455 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:44.696460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:44.696825 (+ 365us) server_negotiation.cc:407] Connection header received
0925 15:09:44.697011 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:44.697015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:44.697067 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:44.697156 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:44.698002 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:44.698006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:44.698009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:09:44.698023 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:44.698060 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:09:44.698074 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:44.698166 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:44.698270 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 15:09:44.698327 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:09:44.699502 13355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad: No bootstrap required, opened a new log
I20260925 15:09:44.707093 13355 raft_consensus.cc:348] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f178a398df1142308cc15390d1e15aad" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38553 } }
I20260925 15:09:44.707460 13355 raft_consensus.cc:374] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:44.707551 13355 raft_consensus.cc:729] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f178a398df1142308cc15390d1e15aad, State: Initialized, Role: FOLLOWER
I20260925 15:09:44.708082 13355 consensus_queue.cc:260] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f178a398df1142308cc15390d1e15aad" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38553 } }
I20260925 15:09:44.708356 13355 raft_consensus.cc:388] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:44.708491 13355 raft_consensus.cc:482] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:44.708635 13355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:44.710953 13355 raft_consensus.cc:504] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f178a398df1142308cc15390d1e15aad" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38553 } }
I20260925 15:09:44.711457 13355 leader_election.cc:302] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f178a398df1142308cc15390d1e15aad; no voters:
I20260925 15:09:44.712007 13355 leader_election.cc:288] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:44.712126 13365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:44.712730 13365 raft_consensus.cc:686] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [term 1 LEADER]: Becoming Leader. State: Replica: f178a398df1142308cc15390d1e15aad, State: Running, Role: LEADER
I20260925 15:09:44.713240 13365 consensus_queue.cc:237] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f178a398df1142308cc15390d1e15aad" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38553 } }
I20260925 15:09:44.713946 13355 sys_catalog.cc:564] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:44.715600 13366 sys_catalog.cc:455] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f178a398df1142308cc15390d1e15aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f178a398df1142308cc15390d1e15aad" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38553 } } }
I20260925 15:09:44.716029 13366 sys_catalog.cc:458] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:44.715718 13367 sys_catalog.cc:455] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [sys.catalog]: SysCatalogTable state changed. Reason: New leader f178a398df1142308cc15390d1e15aad. Latest consensus state: current_term: 1 leader_uuid: "f178a398df1142308cc15390d1e15aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f178a398df1142308cc15390d1e15aad" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 38553 } } }
I20260925 15:09:44.716313 13367 sys_catalog.cc:458] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:44.716892 13373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:44.722322 13373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:44.731930 13373 catalog_manager.cc:1347] Generated new cluster ID: 2a9f59ed199c44998b541d63453e3daf
I20260925 15:09:44.732323 13373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:44.766278 13373 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:44.766765 13373 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:44.779789 13373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f178a398df1142308cc15390d1e15aad: Generated new TSK 0
I20260925 15:09:44.780875 13373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:44.789917 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 13287
W20260925 15:09:44.790933 13350 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45455
W20260925 15:09:44.791589 23472 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 13287: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:09:44.791754 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 13287
2026-09-25T15:09:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1748 ms)
[----------] 18 tests from AuthTokenIssuingTest (32110 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:44 dist-test-slave-nsrc krb5kdc[13386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:44 dist-test-slave-nsrc krb5kdc[13386](info): set up 2 sockets
Sep 25 15:09:44 dist-test-slave-nsrc krb5kdc[13386](info): commencing operation
krb5kdc: starting...
W20260925 15:09:45.804713 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:45 dist-test-slave-nsrc krb5kdc[13386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:09:46.033110 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:36281
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:37809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:36281,127.22.236.61:36549,127.22.236.60:44607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:46.337086 13399 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:09:46.337373 13399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:46.337440 13399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:46.337517 13399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:46.344539 13399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:46.344622 13399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:46.344676 13399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:46.344739 13399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:46.344799 13399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:46.354234 13399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:37809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:36281,127.22.236.61:36549,127.22.236.60:44607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:36281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:46.355372 13399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:46.356591 13399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:46.367520 13407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:46.367482 13405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:46.367499 13404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:46.368289 13399 server_base.cc:1034] running on GCE node
I20260925 15:09:46.368992 13399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:46.370247 13399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:46.371461 13399 hybrid_clock.cc:648] HybridClock initialized: now 1790348986371396 us; error 32 us; skew 500 ppm
I20260925 15:09:46.371742 13399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:46 dist-test-slave-nsrc krb5kdc[13386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:46.374723 13399 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:09:46.376034 13399 webserver.cc:460] Webserver started at http://127.22.236.62:34047/ using document root <none> and password file <none>
I20260925 15:09:46.376727 13399 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:46.376825 13399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:46.377118 13399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:46.379978 13399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "fe8c08c1cc274742978e2ef5e2210ae3"
format_stamp: "Formatted at 2026-09-25 15:09:46 on dist-test-slave-nsrc"
server_key: "2938f5500b46f3f996fb8565f7f84d98"
server_key_iv: "b0540e173b742ffaa9ff1ae6b518979e"
server_key_version: "encryptionkey@0"
I20260925 15:09:46.380777 13399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "fe8c08c1cc274742978e2ef5e2210ae3"
format_stamp: "Formatted at 2026-09-25 15:09:46 on dist-test-slave-nsrc"
server_key: "2938f5500b46f3f996fb8565f7f84d98"
server_key_iv: "b0540e173b742ffaa9ff1ae6b518979e"
server_key_version: "encryptionkey@0"
I20260925 15:09:46.385670 13399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:09:46.388873 13414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:46.390146 13399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:09:46.390300 13399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "fe8c08c1cc274742978e2ef5e2210ae3"
format_stamp: "Formatted at 2026-09-25 15:09:46 on dist-test-slave-nsrc"
server_key: "2938f5500b46f3f996fb8565f7f84d98"
server_key_iv: "b0540e173b742ffaa9ff1ae6b518979e"
server_key_version: "encryptionkey@0"
I20260925 15:09:46.390501 13399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:46.425781 13399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:46.430061 13399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:46.430315 13399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:46.446302 13399 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:36281
I20260925 15:09:46.446321 13465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:36281 every 8 connection(s)
I20260925 15:09:46.447568 13399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:46.450951 13466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:46.452328 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 13399
I20260925 15:09:46.452503 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:46.453131 23472 external_mini_cluster.cc:1428] Setting key 0312df7a216cd9d3bcd1af4fddd267b2
I20260925 15:09:46.457057 13466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:09:46 dist-test-slave-nsrc krb5kdc[13386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.236.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.61@KRBTEST.COM" created.
I20260925 15:09:46.478946 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.460031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36281 (local address 127.0.0.1:59866)
0925 15:09:46.461179 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.461218 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:46.462870 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:46.463766 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:46.463770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:46.464378 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:46.464722 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:46.464738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:46.466791 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:46.466795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:46.467859 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:46.467871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:46.468972 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:46.468977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:46.469092 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:46.469683 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:46.469706 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:46.472238 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:46.475242 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:46.475251 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:46.475260 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:46.475624 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:46.476074 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:46.476077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:46.476080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:46.476257 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:46.477076 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:46.477087 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:46.477311 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 15:09:46.477544 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 15:09:46.477934 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"thread_start_us":677,"threads_started":1}
I20260925 15:09:46.478986 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.460031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.22.236.62:36281)
0925 15:09:46.460808 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:46.460822 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:46.461343 (+ 521us) server_negotiation.cc:407] Connection header received
0925 15:09:46.463247 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:46.463254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:46.463537 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:46.463736 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:46.464930 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:46.466642 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:46.467992 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:46.468755 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:46.468943 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:46.472455 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:46.472464 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:46.472476 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:46.472508 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:46.475126 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:46.475779 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:46.475787 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:46.475789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:46.475982 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:46.476342 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:46.476345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:46.476347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:46.476945 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:46.477135 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:46.477718 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 15:09:46.478084 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":567,"thread_start_us":365,"threads_started":1}
I20260925 15:09:46.486645 13466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:09:46.487807 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.487379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:36549 (local address 127.0.0.1:59802)
0925 15:09:46.487554 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.487670 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 15:09:46.488086 13418 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
I20260925 15:09:46.489658 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.489260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:36549 (local address 127.0.0.1:59808)
0925 15:09:46.489409 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.489487 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:09:46.489871 13466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:36549: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.61.keytab.
Entry for principal kudu/127.22.236.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.61.keytab.
I20260925 15:09:46.504951 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.61:36549
--webserver_interface=127.22.236.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.61.keytab
--principal=kudu/127.22.236.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:37809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:36281,127.22.236.61:36549,127.22.236.60:44607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:09:46.539223 13466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } attempt: 1
I20260925 15:09:46.540498 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.539937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:36549 (local address 127.0.0.1:59820)
0925 15:09:46.540177 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.540328 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 15:09:46.541476 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.541107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:36549 (local address 127.0.0.1:59830)
0925 15:09:46.541247 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.541329 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:09:46.541674 13466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:36549: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
I20260925 15:09:46.609958 13466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } attempt: 2
I20260925 15:09:46.611217 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.610642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:36549 (local address 127.0.0.1:59840)
0925 15:09:46.610913 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.611031 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 15:09:46.612272 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.611838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:36549 (local address 127.0.0.1:59844)
0925 15:09:46.611982 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.612078 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 15:09:46.612457 13466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:36549: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
I20260925 15:09:46.703709 13466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } attempt: 3
I20260925 15:09:46.704909 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.704401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:36549 (local address 127.0.0.1:59848)
0925 15:09:46.704608 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.704738 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 15:09:46.705889 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.705506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:36549 (local address 127.0.0.1:59860)
0925 15:09:46.705643 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.705725 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 15:09:46.706073 13466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:36549: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
W20260925 15:09:46.813935 13474 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:09:46.814234 13474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:46.814301 13474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:46.814381 13474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:46.822055 13474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:46.822146 13474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:46.822203 13474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:46.822253 13474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:46.822312 13474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:46.831830 13474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:37809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:36281,127.22.236.61:36549,127.22.236.60:44607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.61.keytab
--principal=kudu/127.22.236.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.61:36549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.236.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:46.833076 13474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:46.834299 13474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:46.845271 13479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:46.845310 13480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:46.845346 13482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:46.846325 13474 server_base.cc:1034] running on GCE node
I20260925 15:09:46.847074 13474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:46.848413 13474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:46.849289 13466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } attempt: 4
I20260925 15:09:46.849622 13474 hybrid_clock.cc:648] HybridClock initialized: now 1790348986849551 us; error 35 us; skew 500 ppm
I20260925 15:09:46.849938 13474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:09:46.850397 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.849814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:36549 (local address 127.0.0.1:47476)
0925 15:09:46.850132 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.850234 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260925 15:09:46.851351 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.851008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:36549 (local address 127.0.0.1:47482)
0925 15:09:46.851151 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.851233 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:09:46.851553 13466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:36549: Network error: Client connection negotiation failed: client connection to 127.22.236.61:36549: connect: Connection refused (error 111)
Sep 25 15:09:46 dist-test-slave-nsrc krb5kdc[13386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:46.853235 13474 init.cc:377] Logged in from keytab as kudu/127.22.236.61@KRBTEST.COM (short username kudu)
I20260925 15:09:46.854530 13474 webserver.cc:460] Webserver started at http://127.22.236.61:33319/ using document root <none> and password file <none>
I20260925 15:09:46.855171 13474 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:46.855269 13474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:46.855558 13474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:46.858387 13474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/data/instance:
uuid: "95d05f38a8a2444484ba5b0397fad482"
format_stamp: "Formatted at 2026-09-25 15:09:46 on dist-test-slave-nsrc"
server_key: "997da0ea07d7cc5f666fa875f706e229"
server_key_iv: "9f0ee5e1fbc392596e0cf48f40a32e7a"
server_key_version: "encryptionkey@0"
I20260925 15:09:46.859126 13474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/wal/instance:
uuid: "95d05f38a8a2444484ba5b0397fad482"
format_stamp: "Formatted at 2026-09-25 15:09:46 on dist-test-slave-nsrc"
server_key: "997da0ea07d7cc5f666fa875f706e229"
server_key_iv: "9f0ee5e1fbc392596e0cf48f40a32e7a"
server_key_version: "encryptionkey@0"
I20260925 15:09:46.863867 13474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260925 15:09:46.867110 13489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:46.868415 13474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:46.868564 13474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/wal
uuid: "95d05f38a8a2444484ba5b0397fad482"
format_stamp: "Formatted at 2026-09-25 15:09:46 on dist-test-slave-nsrc"
server_key: "997da0ea07d7cc5f666fa875f706e229"
server_key_iv: "9f0ee5e1fbc392596e0cf48f40a32e7a"
server_key_version: "encryptionkey@0"
I20260925 15:09:46.868746 13474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:46.882831 13474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:46.886871 13474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:46.887104 13474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:46.902736 13474 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.61:36549
I20260925 15:09:46.902760 13540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.61:36549 every 8 connection(s)
I20260925 15:09:46.903998 13474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/data/info.pb
I20260925 15:09:46.905110 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 13474
I20260925 15:09:46.905261 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-1/wal/instance
I20260925 15:09:46.905689 23472 external_mini_cluster.cc:1428] Setting key b3578ac02dfde6754c45825fdd2cc803
I20260925 15:09:46.907639 13541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:46.914477 13541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.236.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.60@KRBTEST.COM" created.
Sep 25 15:09:46 dist-test-slave-nsrc krb5kdc[13386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.61@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:09:46.935262 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.917440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.22.236.62:36281)
0925 15:09:46.917645 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:46.917651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:46.918819 (+ 1168us) server_negotiation.cc:407] Connection header received
0925 15:09:46.920670 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:46.920676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:46.920757 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:46.920866 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:46.922241 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:46.923216 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:46.924761 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:46.925383 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:46.925521 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:46.930087 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:46.930092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:46.930095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:46.930126 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:46.932580 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:46.933239 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:46.933242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:46.933245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:46.933355 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:46.933717 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:46.933721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:46.933722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:46.933979 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:46.934114 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:46.934733 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 15:09:46.934946 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:09:46.936184 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.917904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36281 (local address 127.0.0.1:37484)
0925 15:09:46.918655 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.918703 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:46.920417 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:46.921042 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:46.921050 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:46.921550 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:46.922076 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:46.922092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:46.923358 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:46.923362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:46.924600 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:46.924613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:46.925561 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:46.925566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:46.925747 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:46.926331 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:46.926351 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:46.929887 (+ 3536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:46.932700 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:46.932716 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:46.932731 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:46.933116 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:46.933450 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:46.933455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:46.933459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:46.933610 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:46.934101 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:46.934108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:46.934401 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 15:09:46.934614 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 15:09:46.935071 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":308,"threads_started":1}
I20260925 15:09:46.937793 13541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:09:46 dist-test-slave-nsrc krb5kdc[13386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.61@KRBTEST.COM for kudu/127.22.236.61@KRBTEST.COM
I20260925 15:09:46.952040 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.938535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:36549 (local address 127.0.0.1:47488)
0925 15:09:46.938723 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.938741 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:46.938840 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:46.939680 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:46.939684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:46.940087 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:46.940273 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:46.940283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:46.941595 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:46.941599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:46.942848 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:46.942861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:46.943618 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:46.943623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:46.943836 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:46.944426 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:46.944447 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:46.946733 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:46.949668 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:46.949674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:46.949678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:46.950063 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:46.950393 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:46.950398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:46.950401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:46.950563 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:46.951132 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:46.951136 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:46.951272 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 15:09:46.951395 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 15:09:46.951643 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 15:09:46.952206 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.938703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.22.236.61:36549)
0925 15:09:46.939291 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:46.939296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:46.939322 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:09:46.939361 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:46.939367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:46.939537 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:46.939646 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:46.940422 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:46.941467 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:46.942974 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:46.943487 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:46.943571 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:46.946916 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:46.946926 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:46.946940 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:46.946967 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:46.949556 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:46.950181 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:46.950188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:46.950190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:46.950295 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:46.950658 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:46.950663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:46.950666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:46.951025 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:46.951214 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:46.951571 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 15:09:46.951778 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":322,"threads_started":1}
I20260925 15:09:46.954367 13541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:09:46.955394 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.954964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:37988)
0925 15:09:46.955133 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.955244 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.60.keytab.
Entry for principal kudu/127.22.236.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.60.keytab.
W20260925 15:09:46.955629 13490 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
I20260925 15:09:46.956754 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.60:44607
--webserver_interface=127.22.236.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.60.keytab
--principal=kudu/127.22.236.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:37809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:36281,127.22.236.61:36549,127.22.236.60:44607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:09:46.957132 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:46.956775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:37998)
0925 15:09:46.956919 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:46.957022 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 15:09:46.957319 13541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:44607: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
I20260925 15:09:47.006592 13541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } attempt: 1
I20260925 15:09:47.007735 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.007295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38010)
0925 15:09:47.007479 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.007586 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 15:09:47.008651 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.008317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38016)
0925 15:09:47.008467 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.008543 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:09:47.008839 13541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:44607: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
I20260925 15:09:47.077026 13541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } attempt: 2
I20260925 15:09:47.078136 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.077692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38028)
0925 15:09:47.077868 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.077981 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 15:09:47.079079 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.078743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38044)
0925 15:09:47.078886 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.078965 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:09:47.079273 13541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:44607: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
I20260925 15:09:47.150770 13466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } attempt: 5
Sep 25 15:09:47 dist-test-slave-nsrc krb5kdc[13386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for kudu/127.22.236.61@KRBTEST.COM
I20260925 15:09:47.163511 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.151509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:36549 (local address 127.0.0.1:47498)
0925 15:09:47.151711 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.151732 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:47.151861 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:47.152224 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:47.152230 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:47.152638 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:47.152831 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:47.152840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.153804 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.153809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:47.154709 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:47.154721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.155443 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.155449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:47.155659 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.156317 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:47.156334 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:47.158429 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:47.161255 (+ 2826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:47.161261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:47.161265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:47.161613 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:47.161948 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:47.161953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:47.161957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:47.162084 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:47.162574 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:47.162578 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:47.162722 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 15:09:47.162849 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 15:09:47.163123 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 15:09:47.163507 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.151573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.22.236.61:36549)
0925 15:09:47.151758 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.151764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.151788 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:09:47.151992 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.151996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.152072 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.152178 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.152984 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.153675 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.154862 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.155302 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.155413 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.158623 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:47.158629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:47.158632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:47.158665 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:47.161135 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.161722 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.161726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.161728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.161847 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.162179 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.162183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.162186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.162467 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:47.162594 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:47.162965 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 15:09:47.163170 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 15:09:47.164788 13466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:09:47.165784 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.165420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38050)
0925 15:09:47.165580 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.165654 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 15:09:47.166610 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.166302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38062)
0925 15:09:47.166434 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.166511 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:09:47.166779 13466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:44607: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
I20260925 15:09:47.170457 13541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } attempt: 3
I20260925 15:09:47.171264 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.170885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38074)
0925 15:09:47.171064 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.171140 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 15:09:47.172227 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.171794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38086)
0925 15:09:47.171998 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.172065 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260925 15:09:47.172435 13541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:44607: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
I20260925 15:09:47.217981 13466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } attempt: 1
I20260925 15:09:47.219035 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.218618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38090)
0925 15:09:47.218791 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.218874 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 15:09:47.219921 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.219614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38094)
0925 15:09:47.219751 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.219817 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:09:47.220109 13466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:44607: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
W20260925 15:09:47.269033 13548 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:09:47.269357 13548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:47.269457 13548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:47.269569 13548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:47.276731 13548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:47.276821 13548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:47.276904 13548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:47.276968 13548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:47.277061 13548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:47.286792 13548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:37809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:36281,127.22.236.61:36549,127.22.236.60:44607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.60.keytab
--principal=kudu/127.22.236.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.60:44607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.236.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:47.288010 13548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:47.288319 13466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } attempt: 2
I20260925 15:09:47.289288 13548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:09:47.289342 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.288892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38096)
0925 15:09:47.289104 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.289186 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 15:09:47.290369 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.289994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38104)
0925 15:09:47.290145 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.290236 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:09:47.290591 13466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:44607: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
W20260925 15:09:47.300653 13553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:47.300616 13556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:47.300562 13554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:47.301443 13548 server_base.cc:1034] running on GCE node
I20260925 15:09:47.302151 13548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:47.303458 13548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:47.304658 13548 hybrid_clock.cc:648] HybridClock initialized: now 1790348987304592 us; error 44 us; skew 500 ppm
I20260925 15:09:47.304973 13548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:47 dist-test-slave-nsrc krb5kdc[13386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348987, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:47.307734 13548 init.cc:377] Logged in from keytab as kudu/127.22.236.60@KRBTEST.COM (short username kudu)
I20260925 15:09:47.309044 13548 webserver.cc:460] Webserver started at http://127.22.236.60:42939/ using document root <none> and password file <none>
I20260925 15:09:47.309669 13548 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:47.309759 13548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:47.310045 13548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:47.312860 13548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/data/instance:
uuid: "9ee58ffb9d9549d8bbba891395a1f9cf"
format_stamp: "Formatted at 2026-09-25 15:09:47 on dist-test-slave-nsrc"
server_key: "01105bd22178c6f87b05bce72b8f19ad"
server_key_iv: "c694c357bdd896d9e49facc813682789"
server_key_version: "encryptionkey@0"
I20260925 15:09:47.313591 13548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/wal/instance:
uuid: "9ee58ffb9d9549d8bbba891395a1f9cf"
format_stamp: "Formatted at 2026-09-25 15:09:47 on dist-test-slave-nsrc"
server_key: "01105bd22178c6f87b05bce72b8f19ad"
server_key_iv: "c694c357bdd896d9e49facc813682789"
server_key_version: "encryptionkey@0"
I20260925 15:09:47.315625 13541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } attempt: 4
I20260925 15:09:47.316615 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.316127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38114)
0925 15:09:47.316369 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.316465 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 15:09:47.317544 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.317179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38118)
0925 15:09:47.317356 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.317442 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 15:09:47.317752 13541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:44607: Network error: Client connection negotiation failed: client connection to 127.22.236.60:44607: connect: Connection refused (error 111)
I20260925 15:09:47.318457 13548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:09:47.321573 13563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:47.322753 13548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:09:47.322893 13548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/wal
uuid: "9ee58ffb9d9549d8bbba891395a1f9cf"
format_stamp: "Formatted at 2026-09-25 15:09:47 on dist-test-slave-nsrc"
server_key: "01105bd22178c6f87b05bce72b8f19ad"
server_key_iv: "c694c357bdd896d9e49facc813682789"
server_key_version: "encryptionkey@0"
I20260925 15:09:47.323071 13548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:47.359071 13548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:47.362753 13548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:47.362991 13548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:47.379552 13548 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.60:44607
I20260925 15:09:47.379575 13615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.60:44607 every 8 connection(s)
I20260925 15:09:47.380872 13548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/data/info.pb
I20260925 15:09:47.384313 13616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:47.387554 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 13548
I20260925 15:09:47.387701 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348645297121-23472-0/minicluster-data/master-2/wal/instance
I20260925 15:09:47.388113 23472 external_mini_cluster.cc:1428] Setting key 2b3a71f80b52ecd2512f96cd01a53387
I20260925 15:09:47.391287 13616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:09:47.396787 13466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } attempt: 3
Sep 25 15:09:47 dist-test-slave-nsrc krb5kdc[13386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.60@KRBTEST.COM
Sep 25 15:09:47 dist-test-slave-nsrc krb5kdc[13386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348987, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.60@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
Sep 25 15:09:47 dist-test-slave-nsrc krb5kdc[13386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for kudu/127.22.236.60@KRBTEST.COM
I20260925 15:09:47.410611 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.394526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.22.236.62:36281)
0925 15:09:47.394755 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.394761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.395480 (+ 719us) server_negotiation.cc:407] Connection header received
0925 15:09:47.395645 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.395649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.395711 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.395788 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.397218 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.398258 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.400040 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.400583 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.400706 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.404568 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:47.404573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:47.404576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:47.404606 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:47.407427 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.408197 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.408202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.408205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.408371 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.408792 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.408797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.408801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.409098 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:47.409231 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:47.410007 (+ 776us) server_negotiation.cc:299] Negotiation successful
0925 15:09:47.410234 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:09:47.411727 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.394556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:36281 (local address 127.0.0.1:37492)
0925 15:09:47.395337 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.395383 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:47.395539 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:47.395873 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:47.395885 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:47.396739 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:47.397067 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:47.397079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.398409 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.398414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:47.399864 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:47.399877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.400778 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.400785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:47.400954 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.401614 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:47.401645 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:47.404392 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:47.407527 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:47.407539 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:47.407551 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:47.408080 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:47.408468 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:47.408473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:47.408477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:47.408681 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:47.409222 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:47.409229 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:47.409556 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 15:09:47.409862 (+ 306us) client_negotiation.cc:240] Negotiation successful
0925 15:09:47.410321 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":2,"thread_start_us":386,"threads_started":1}
I20260925 15:09:47.412268 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.389836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.22.236.60:44607)
0925 15:09:47.390536 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.390547 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.390590 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:09:47.392207 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.392228 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.392651 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.392995 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.394576 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.396552 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.398200 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.399330 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.399503 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.405288 (+ 5785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:47.405300 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:47.405316 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:47.405350 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:47.408863 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.409682 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.409689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.409692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.409821 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.410251 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.410256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.410259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.410684 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:47.410971 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:47.411497 (+ 526us) server_negotiation.cc:299] Negotiation successful
0925 15:09:47.411733 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":499,"thread_start_us":285,"threads_started":1}
I20260925 15:09:47.412568 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.397366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38134)
0925 15:09:47.397636 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.397654 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:47.397743 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:47.398484 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:47.398488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:47.398963 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:47.399144 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:47.399156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.400635 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.400639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:47.401741 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:47.401753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.402438 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.402443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:47.402626 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.403096 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:47.403113 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:47.405353 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:47.409700 (+ 4347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:47.409704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:47.409708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:47.410170 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:47.410569 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:47.410574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:47.410577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:47.410909 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:47.411498 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:47.411503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:47.411667 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 15:09:47.411799 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 15:09:47.412219 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260925 15:09:47.413589 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.397379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.22.236.60:44607)
0925 15:09:47.398199 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.398203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.398219 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:09:47.398253 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.398256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.398304 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.398454 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.399327 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.400510 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.401870 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.402296 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.402397 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.405578 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:47.405584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:47.405589 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:47.405620 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:47.409587 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.410300 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.410304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.410307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.410447 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.411044 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.411048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.411051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.411361 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:47.412895 (+ 1534us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:47.413026 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 15:09:47.413227 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":701,"thread_start_us":544,"threads_started":1}
I20260925 15:09:47.413614 13616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:09:47.419142 13466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3: Bootstrap starting.
Sep 25 15:09:47 dist-test-slave-nsrc krb5kdc[13386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348987, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.60@KRBTEST.COM for kudu/127.22.236.61@KRBTEST.COM
I20260925 15:09:47.423079 13466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:47.424355 13466 log.cc:824] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:09:47 dist-test-slave-nsrc krb5kdc[13386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:09:47.427452 13466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3: No bootstrap required, opened a new log
I20260925 15:09:47.428638 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.414371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:36549 (local address 127.0.0.1:47514)
0925 15:09:47.415301 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.415318 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:47.415447 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:47.415774 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:47.415778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:47.416285 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:47.416585 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:47.416598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.417966 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.417971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:47.419047 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:47.419059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.419785 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.419791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:47.419964 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.420560 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:47.420581 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:47.422642 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:47.425849 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:47.425854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:47.425859 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:47.426260 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:47.426716 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:47.426721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:47.426724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:47.426881 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:47.427638 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:47.427643 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:47.427798 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 15:09:47.427947 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 15:09:47.428233 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":611,"threads_started":1}
I20260925 15:09:47.428673 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.414316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.22.236.61:36549)
0925 15:09:47.414526 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.414533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.415415 (+ 882us) server_negotiation.cc:407] Connection header received
0925 15:09:47.415556 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.415561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.415638 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.415738 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.416747 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.417785 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.419179 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.419646 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.419753 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.422814 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:47.422819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:47.422823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:47.422852 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:47.425740 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.426376 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.426381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.426384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.426595 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.427015 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.427020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.427023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.427367 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:47.427513 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:47.428075 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 15:09:47.428311 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:09:47.430006 13616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:09:47.432336 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.415428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.22.236.62:36281)
0925 15:09:47.415897 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.415902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.415918 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:09:47.415981 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.415986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.416058 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.416177 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.417477 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.418398 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.419952 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.420544 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.420665 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.425626 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:47.425631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:47.425635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:47.425667 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:47.428961 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.429781 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.429784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.429787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.429959 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.430334 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.430340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.430343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.430736 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:47.431022 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:47.431801 (+ 779us) server_negotiation.cc:299] Negotiation successful
0925 15:09:47.431998 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356}
I20260925 15:09:47.436872 13466 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } }
I20260925 15:09:47.437238 13466 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8c08c1cc274742978e2ef5e2210ae3, State: Initialized, Role: FOLLOWER
I20260925 15:09:47.437654 13466 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } }
Sep 25 15:09:47 dist-test-slave-nsrc krb5kdc[13386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348987, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.60@KRBTEST.COM for kudu/127.22.236.60@KRBTEST.COM
I20260925 15:09:47.438493 13624 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } } }
I20260925 15:09:47.438922 13624 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:09:47.439679 13466 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 15:09:47 dist-test-slave-nsrc krb5kdc[13386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.61@KRBTEST.COM
I20260925 15:09:47.444234 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.430702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38142)
0925 15:09:47.431250 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.431267 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:47.431361 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:47.431765 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:47.431769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:47.432087 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:47.432270 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:47.432278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.433468 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.433473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:47.434580 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:47.434594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.435599 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.435605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:47.435755 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.436340 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:47.436357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:47.438479 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:47.442026 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:47.442030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:47.442034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:47.442505 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:47.442866 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:47.442870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:47.442873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:47.443026 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:47.443514 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:47.443519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:47.443663 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 15:09:47.443779 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 15:09:47.443982 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438}
I20260925 15:09:47.444955 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.431085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.22.236.60:44607)
0925 15:09:47.431250 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.431255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.431309 (+ 54us) server_negotiation.cc:407] Connection header received
0925 15:09:47.431457 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.431462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.431517 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.431585 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.432386 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.433219 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.434713 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.435308 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.435400 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.438659 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:47.438665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:47.438669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:47.438702 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:47.441868 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.442630 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.442634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.442638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.442759 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.443131 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.443135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.443139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.443404 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:47.443544 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:47.444497 (+ 953us) server_negotiation.cc:299] Negotiation successful
0925 15:09:47.444707 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:09:47.450562 13616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf: Bootstrap starting.
I20260925 15:09:47.451737 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.434076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.22.236.61:36549)
0925 15:09:47.434225 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.434231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.434284 (+ 53us) server_negotiation.cc:407] Connection header received
0925 15:09:47.434433 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.434438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.434503 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.434596 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.435988 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.436970 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.438659 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.439247 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.439335 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.444908 (+ 5573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:47.444915 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:47.444919 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:47.444965 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:47.448182 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.448921 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.448926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.448930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.449088 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.449534 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.449539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.449542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.449865 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:47.451086 (+ 1221us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:47.451213 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 15:09:47.451425 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:09:47.454833 13616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:47.456342 13616 log.cc:824] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:47.456457 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:47.460072 13441 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fe8c08c1cc274742978e2ef5e2210ae3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37508
I20260925 15:09:47.460523 13441 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:09:47.460714 13441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:47.462973 13616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf: No bootstrap required, opened a new log
I20260925 15:09:47.463881 13441 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } }
I20260925 15:09:47.465363 13441 leader_election.cc:288] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [CANDIDATE]: Term 1 election: Requested vote from peers 95d05f38a8a2444484ba5b0397fad482 (127.22.236.61:36549), 9ee58ffb9d9549d8bbba891395a1f9cf (127.22.236.60:44607)
W20260925 15:09:47.472507 13636 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:09:47.472627 13636 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:09:47.469203 13399 master_runner.cc:186] Error getting master registration for 127.22.236.61:36549: OK, instance_id { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" instance_seqno: 1790348986898265 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:09:47.479950 13399 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:09:47.484911 13616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } }
I20260925 15:09:47.485347 13616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee58ffb9d9549d8bbba891395a1f9cf, State: Initialized, Role: FOLLOWER
I20260925 15:09:47.485858 13616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } }
I20260925 15:09:47.493640 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.476105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.22.236.62:36281)
0925 15:09:47.476294 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.476300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.476317 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:09:47.476369 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.476373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.476467 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.476574 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.478740 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.480004 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.481977 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.482662 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.482768 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.486299 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:47.486306 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:47.486311 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:47.486350 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:47.490024 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.490964 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.490970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.490974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.491170 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.491674 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.491679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.491682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.492077 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:47.492334 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:47.492955 (+ 621us) server_negotiation.cc:299] Negotiation successful
0925 15:09:47.493221 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:09:47.498989 13644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } } }
I20260925 15:09:47.499415 13644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:09:47.501247 13616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:47.501343 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf"
I20260925 15:09:47.501830 13590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:47.506057 13590 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8c08c1cc274742978e2ef5e2210ae3 in term 1.
I20260925 15:09:47.507458 13415 leader_election.cc:302] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ee58ffb9d9549d8bbba891395a1f9cf, fe8c08c1cc274742978e2ef5e2210ae3; no voters:
I20260925 15:09:47.508126 13624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:47.508981 13624 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [term 1 LEADER]: Becoming Leader. State: Replica: fe8c08c1cc274742978e2ef5e2210ae3, State: Running, Role: LEADER
I20260925 15:09:47.509559 13624 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } }
I20260925 15:09:47.521219 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.476353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.22.236.60:44607)
0925 15:09:47.476818 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.476825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.489989 (+ 13164us) server_negotiation.cc:407] Connection header received
0925 15:09:47.490963 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.490968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.491065 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.491141 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.494770 (+ 3629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.495890 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.497894 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.498563 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.498680 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.503347 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:47.503353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:47.503358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:47.503395 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:47.507000 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.518114 (+ 11114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.518120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.518124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.518300 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.519016 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.519021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.519024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.519392 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:47.519644 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:47.520618 (+ 974us) server_negotiation.cc:299] Negotiation successful
0925 15:09:47.520850 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337}
I20260925 15:09:47.523154 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.484718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.22.236.61:36549)
0925 15:09:47.484890 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.484896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.492648 (+ 7752us) server_negotiation.cc:407] Connection header received
0925 15:09:47.493094 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.493100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.493183 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.493251 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.497324 (+ 4073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.498134 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.500494 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.504428 (+ 3934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.504558 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.514505 (+ 9947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:47.514513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:47.514519 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:47.514576 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:47.518215 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.519488 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.519493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.519496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.519644 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.520258 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.520386 (+ 128us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.520390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.520934 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:47.521283 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:47.522226 (+ 943us) server_negotiation.cc:299] Negotiation successful
0925 15:09:47.522609 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:09:47.529784 13647 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe8c08c1cc274742978e2ef5e2210ae3. Latest consensus state: current_term: 1 leader_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } } }
I20260925 15:09:47.530159 13647 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:47.530949 13653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:47.534207 13653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:47.544265 13590 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf [term 1 FOLLOWER]: Refusing update from remote peer fe8c08c1cc274742978e2ef5e2210ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:47.545148 13647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:47.563799 13644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe8c08c1cc274742978e2ef5e2210ae3. Latest consensus state: current_term: 1 leader_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } } }
I20260925 15:09:47.564121 13644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:09:47.576222 13647 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } } }
I20260925 15:09:47.576545 13647 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [sys.catalog]: This master's current role is: LEADER
W20260925 15:09:47.581571 13663 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:09:47.581696 13663 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:09:47.580391 13548 master_runner.cc:186] Error getting master registration for 127.22.236.61:36549: OK, instance_id { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" instance_seqno: 1790348986898265 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:09:47.594166 13653 catalog_manager.cc:1347] Generated new cluster ID: 6683b9b6fb1e42b99307108ff01b8460
I20260925 15:09:47.594272 13653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:47.604802 13548 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:09:47.613046 13644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } } }
I20260925 15:09:47.613329 13644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:09:47.616948 13541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } attempt: 5
Sep 25 15:09:47 dist-test-slave-nsrc krb5kdc[13386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.61@KRBTEST.COM for kudu/127.22.236.60@KRBTEST.COM
I20260925 15:09:47.634853 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.617830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.22.236.60:44607)
0925 15:09:47.618005 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.618011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.618029 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:09:47.618085 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.618090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.618157 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.618243 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.619126 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.620274 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.622120 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.622874 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.622986 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.627077 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:47.627083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:47.627088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:47.627129 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:47.630641 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.631750 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.631756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.631760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.631925 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:47.632597 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:47.632602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:47.632606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:47.632961 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:47.633077 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:47.633878 (+ 801us) server_negotiation.cc:299] Negotiation successful
0925 15:09:47.634221 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:09:47.637166 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:47.617520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:44607 (local address 127.0.0.1:38158)
0925 15:09:47.617708 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:47.617730 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:47.617848 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:47.618286 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:47.618290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:47.618794 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:47.618992 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:47.619005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.620443 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.620449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:47.621983 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:47.621999 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:47.623190 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:47.623195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:47.623417 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:47.624109 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:47.624131 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:47.626890 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:47.630773 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:47.630778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:47.630809 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:47.631582 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:47.632107 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:47.632112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:47.632116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:47.632407 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:47.633213 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:47.633218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:47.633482 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 15:09:47.635798 (+ 2316us) client_negotiation.cc:240] Negotiation successful
0925 15:09:47.636582 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:09:47.639834 13653 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:47.643110 13541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482: Bootstrap starting.
I20260925 15:09:47.644912 13653 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:47.649077 13541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:47.651027 13541 log.cc:824] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:47.663919 13541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482: No bootstrap required, opened a new log
I20260925 15:09:47.664826 13653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3: Generated new TSK 0
I20260925 15:09:47.666555 13653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:47.680948 13541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } }
I20260925 15:09:47.681676 13541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d05f38a8a2444484ba5b0397fad482, State: Initialized, Role: FOLLOWER
I20260925 15:09:47.682509 13541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } }
I20260925 15:09:47.693535 13516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "95d05f38a8a2444484ba5b0397fad482"
I20260925 15:09:47.693872 13515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:47.697433 13515 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482 [term 1 FOLLOWER]: Refusing update from remote peer fe8c08c1cc274742978e2ef5e2210ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:47.698526 13541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:47.698491 13647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:47.703648 13516 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8c08c1cc274742978e2ef5e2210ae3 in term 1.
I20260925 15:09:47.705368 13667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } } }
I20260925 15:09:47.705796 13667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:09:47.732595 13672 mvcc.cc:205] Tried to move back new op lower bound from 7333269452966744064 to 7333269452844175360. Current Snapshot: MvccSnapshot[applied={T|T < 7333269452966744064}]
I20260925 15:09:47.745931 13667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe8c08c1cc274742978e2ef5e2210ae3. Latest consensus state: current_term: 1 leader_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } } }
I20260925 15:09:47.746292 13667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:09:47.745965 13670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } } }
I20260925 15:09:47.746388 13670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:09:47.803673 13688 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260925 15:09:47.831511 13470 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:47.817180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.22.236.62:36281)
0925 15:09:47.817383 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.817390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.829309 (+ 11919us) server_negotiation.cc:407] Connection header received
0925 15:09:47.829827 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.829834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.829926 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.830006 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.831259 (+ 1253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37518: BlockingRecv error: recv got EOF from 127.0.0.1:37518 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 15:09:47.833622 13621 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:47.816960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.22.236.60:44607)
0925 15:09:47.817127 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.817133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.830690 (+ 13557us) server_negotiation.cc:407] Connection header received
0925 15:09:47.831779 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.831785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.831861 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.831932 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.833367 (+ 1435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38160: BlockingRecv error: recv got EOF from 127.0.0.1:38160 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 15:09:47.834342 13546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:47.825228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47552 (local address 127.22.236.61:36549)
0925 15:09:47.825397 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.825404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.831196 (+ 5792us) server_negotiation.cc:407] Connection header received
0925 15:09:47.832127 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.832133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.832242 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.832312 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.834137 (+ 1825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47552: BlockingRecv error: recv got EOF from 127.0.0.1:47552 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 15:09:47.841580 13546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:47.839382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.22.236.61:36549)
0925 15:09:47.839550 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.839555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.839996 (+ 441us) server_negotiation.cc:407] Connection header received
0925 15:09:47.840308 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.840314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.840436 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.840543 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.841323 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47568: BlockingRecv error: recv got EOF from 127.0.0.1:47568 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 15:09:47.854153 13546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:47.845987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.22.236.61:36549)
0925 15:09:47.846138 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.846143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.852418 (+ 6275us) server_negotiation.cc:407] Connection header received
0925 15:09:47.852844 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.852850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.852938 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.853006 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.853958 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47584: BlockingRecv error: recv got EOF from 127.0.0.1:47584 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 15:09:47.871866 13546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:47.869670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.22.236.61:36549)
0925 15:09:47.869844 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.869850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.870369 (+ 519us) server_negotiation.cc:407] Connection header received
0925 15:09:47.870561 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.870568 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.870675 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.870793 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.871604 (+ 811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47596: BlockingRecv error: recv got EOF from 127.0.0.1:47596 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 15:09:47.880759 13546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:47.878754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47600 (local address 127.22.236.61:36549)
0925 15:09:47.878954 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.878959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.879431 (+ 472us) server_negotiation.cc:407] Connection header received
0925 15:09:47.879587 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.879593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.879680 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.879782 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.880531 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47600: BlockingRecv error: recv got EOF from 127.0.0.1:47600 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260925 15:09:47.894622 13546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:47.892459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.22.236.61:36549)
0925 15:09:47.892631 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.892637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.893133 (+ 496us) server_negotiation.cc:407] Connection header received
0925 15:09:47.893339 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.893344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.893439 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.893579 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.894376 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47602: BlockingRecv error: recv got EOF from 127.0.0.1:47602 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 15:09:47.936200 13546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:47.933643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.22.236.61:36549)
0925 15:09:47.933849 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.933856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.934485 (+ 629us) server_negotiation.cc:407] Connection header received
0925 15:09:47.934741 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.934746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.934855 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.934964 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.935949 (+ 985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47612: BlockingRecv error: recv got EOF from 127.0.0.1:47612 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 15:09:47.974102 13546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:47.971711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47624 (local address 127.22.236.61:36549)
0925 15:09:47.971895 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:47.971901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:47.972538 (+ 637us) server_negotiation.cc:407] Connection header received
0925 15:09:47.972743 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:47.972749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:47.972852 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:47.972965 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:47.973847 (+ 882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47624: BlockingRecv error: recv got EOF from 127.0.0.1:47624 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 15:09:48.042973 13546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:48.041111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.22.236.61:36549)
0925 15:09:48.041326 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:48.041332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:48.041636 (+ 304us) server_negotiation.cc:407] Connection header received
0925 15:09:48.041834 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:48.041840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:48.041930 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:48.042019 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:48.042750 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47632: BlockingRecv error: recv got EOF from 127.0.0.1:47632 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260925 15:09:48.175823 13546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:48.174118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.22.236.61:36549)
0925 15:09:48.174289 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:48.174294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:48.174652 (+ 358us) server_negotiation.cc:407] Connection header received
0925 15:09:48.174810 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:48.174814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:48.174882 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:48.174998 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:48.175624 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47644: BlockingRecv error: recv got EOF from 127.0.0.1:47644 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:09:48.258127 13647 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe8c08c1cc274742978e2ef5e2210ae3" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 36281 } } peers { permanent_uuid: "95d05f38a8a2444484ba5b0397fad482" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 36549 } } peers { permanent_uuid: "9ee58ffb9d9549d8bbba891395a1f9cf" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 44607 } } }
I20260925 15:09:48.258431 13647 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe8c08c1cc274742978e2ef5e2210ae3 [sys.catalog]: This master's current role is: LEADER
W20260925 15:09:48.436754 13546 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:48.435128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.22.236.61:36549)
0925 15:09:48.435303 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:48.435307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:48.435608 (+ 301us) server_negotiation.cc:407] Connection header received
0925 15:09:48.435769 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:48.435774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:48.435842 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:48.435942 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:48.436591 (+ 649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47654: BlockingRecv error: recv got EOF from 127.0.0.1:47654 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:09:48.583526 13663 catalog_manager.cc:1259] Loaded cluster ID: 6683b9b6fb1e42b99307108ff01b8460
I20260925 15:09:48.583626 13663 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf: loading cluster ID for follower catalog manager: success
I20260925 15:09:48.586138 13663 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf: acquiring CA information for follower catalog manager: success
I20260925 15:09:48.587909 13663 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ee58ffb9d9549d8bbba891395a1f9cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:09:48.807859 13688 catalog_manager.cc:1259] Loaded cluster ID: 6683b9b6fb1e42b99307108ff01b8460
I20260925 15:09:48.807963 13688 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482: loading cluster ID for follower catalog manager: success
I20260925 15:09:48.810483 13688 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482: acquiring CA information for follower catalog manager: success
I20260925 15:09:48.812213 13688 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 95d05f38a8a2444484ba5b0397fad482: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:09:48.969151 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:48.962925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47658 (local address 127.22.236.61:36549)
0925 15:09:48.963712 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:48.963718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:48.963735 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:09:48.963800 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:48.963803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:48.963953 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:48.964059 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:48.964572 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:48.965407 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:48.966711 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:48.967271 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:48.967385 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:48.967986 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:48.968569 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:48.968689 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:48.968794 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 15:09:48.968858 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":509,"threads_started":1}
I20260925 15:09:48.981278 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:48.975404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.22.236.60:44607)
0925 15:09:48.975947 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:48.975951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:48.975967 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:09:48.976058 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:48.976061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:48.976234 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:48.976363 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:48.976733 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:48.977424 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:48.979216 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:48.979593 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:48.979675 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:48.980202 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:48.980724 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:48.980826 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:48.980937 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 15:09:48.981000 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":359,"threads_started":1}
I20260925 15:09:49.005684 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:48.988334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.22.236.62:36281)
0925 15:09:48.998265 (+ 9931us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:48.998272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:48.998295 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:09:48.998419 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:48.998425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:48.998638 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:48.998818 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:48.999445 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:49.000630 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:49.002648 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:49.003240 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:49.003357 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:49.004085 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:49.004573 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:49.005217 (+ 644us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:49.005268 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 15:09:49.005353 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9794,"thread_start_us":10603,"threads_started":1}
I20260925 15:09:49.006623 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:48.987938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.22.236.61:36549)
0925 15:09:48.988599 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:48.988605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:48.999534 (+ 10929us) server_negotiation.cc:407] Connection header received
0925 15:09:48.999784 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:48.999790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:48.999869 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:48.999947 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:49.000693 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:49.001814 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:49.004559 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:49.005199 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:49.005292 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:49.005872 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:49.006007 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:49.006081 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:49.006282 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 15:09:49.006363 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536}
I20260925 15:09:49.010601 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 13399
I20260925 15:09:49.011322 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:48.988120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.22.236.60:44607)
0925 15:09:48.988284 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:48.988289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:49.004159 (+ 15870us) server_negotiation.cc:407] Connection header received
0925 15:09:49.004304 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:49.004310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:49.004402 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:49.004496 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:49.005207 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:49.006286 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:49.009460 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:49.010082 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:49.010201 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:49.010644 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:49.010764 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:49.010844 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:49.010974 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 15:09:49.011055 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:09:49.011554 13461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:49.122816 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 13399
W20260925 15:09:49.139138 13564 connection.cc:535] server connection from 127.0.0.1:38134 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:09:49.139555 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 13474
I20260925 15:09:49.140628 13536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:49.252322 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 13474
I20260925 15:09:49.267818 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 13548
I20260925 15:09:49.268923 13611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:49.373927 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 13548
2026-09-25T15:09:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4590 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:49 dist-test-slave-nsrc krb5kdc[13774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:49 dist-test-slave-nsrc krb5kdc[13774](info): set up 2 sockets
Sep 25 15:09:49 dist-test-slave-nsrc krb5kdc[13774](info): commencing operation
krb5kdc: starting...
W20260925 15:09:50.396611 23472 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/test-user.keytab.
Sep 25 15:09:50 dist-test-slave-nsrc krb5kdc[13774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.236.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
Entry for principal kudu/127.22.236.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab.
I20260925 15:09:50.635218 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:42095
--webserver_interface=127.22.236.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:42095,127.22.236.61:40375,127.22.236.60:41031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:50.947046 13787 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:09:50.947391 13787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:50.947489 13787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:50.947618 13787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:50.954800 13787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:50.954900 13787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:50.954977 13787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:50.955041 13787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:50.955123 13787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:50.964991 13787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:42095,127.22.236.61:40375,127.22.236.60:41031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.62.keytab
--principal=kudu/127.22.236.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.62:42095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.236.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:50.966297 13787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:50.967571 13787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:50.978782 13792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:50.978849 13793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:50.978828 13795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:50.979661 13787 server_base.cc:1034] running on GCE node
I20260925 15:09:50.980446 13787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:50.981825 13787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:50.983032 13787 hybrid_clock.cc:648] HybridClock initialized: now 1790348990982979 us; error 36 us; skew 500 ppm
I20260925 15:09:50.983348 13787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:50 dist-test-slave-nsrc krb5kdc[13774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348990, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:50.991451 13787 init.cc:377] Logged in from keytab as kudu/127.22.236.62@KRBTEST.COM (short username kudu)
I20260925 15:09:50.993024 13787 webserver.cc:460] Webserver started at http://127.22.236.62:46193/ using document root <none> and password file <none>
I20260925 15:09:50.993690 13787 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:50.993798 13787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:50.994088 13787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:50.997056 13787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/data/instance:
uuid: "1a436229ec8243879f31021f400d2f83"
format_stamp: "Formatted at 2026-09-25 15:09:50 on dist-test-slave-nsrc"
server_key: "f5dc9d631c23981c30cc1f2e1c13480e"
server_key_iv: "ae725ce6a8e39e451fbe9c90784c8504"
server_key_version: "encryptionkey@0"
I20260925 15:09:50.997816 13787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/wal/instance:
uuid: "1a436229ec8243879f31021f400d2f83"
format_stamp: "Formatted at 2026-09-25 15:09:50 on dist-test-slave-nsrc"
server_key: "f5dc9d631c23981c30cc1f2e1c13480e"
server_key_iv: "ae725ce6a8e39e451fbe9c90784c8504"
server_key_version: "encryptionkey@0"
I20260925 15:09:51.002810 13787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 15:09:51.006587 13802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:51.008020 13787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:09:51.008216 13787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/wal
uuid: "1a436229ec8243879f31021f400d2f83"
format_stamp: "Formatted at 2026-09-25 15:09:50 on dist-test-slave-nsrc"
server_key: "f5dc9d631c23981c30cc1f2e1c13480e"
server_key_iv: "ae725ce6a8e39e451fbe9c90784c8504"
server_key_version: "encryptionkey@0"
I20260925 15:09:51.008428 13787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:51.021310 13787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:51.025233 13787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:51.025485 13787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:51.042286 13787 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.62:42095
I20260925 15:09:51.042306 13853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.62:42095 every 8 connection(s)
I20260925 15:09:51.043535 13787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:51.045850 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 13787
I20260925 15:09:51.046015 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-0/wal/instance
I20260925 15:09:51.046422 23472 external_mini_cluster.cc:1428] Setting key dff6b7493609b2361ae6350436396224
I20260925 15:09:51.047104 13854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:51.053498 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.236.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.61@KRBTEST.COM" created.
Sep 25 15:09:51 dist-test-slave-nsrc krb5kdc[13774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348990, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:09:51.077520 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.057522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.22.236.62:42095)
0925 15:09:51.058236 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:51.058243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:51.058299 (+ 56us) server_negotiation.cc:407] Connection header received
0925 15:09:51.059560 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:51.059568 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:51.059872 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:51.060071 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:51.061195 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.062953 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.064318 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.065186 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.065360 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:51.070928 (+ 5568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:51.070935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:51.070947 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:51.070980 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:51.073558 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:51.074186 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:51.074192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:51.074194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:51.074345 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:51.074695 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:51.074698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:51.074707 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:51.075170 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:51.075353 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:51.075841 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 15:09:51.076107 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":381,"threads_started":1}
I20260925 15:09:51.077549 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.056333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:42095 (local address 127.0.0.1:53896)
0925 15:09:51.057104 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.057149 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:51.059189 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:51.060109 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:51.060115 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:51.060663 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:51.060992 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:51.061009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.063118 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.063122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:51.064185 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:51.064199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.065498 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.065508 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:51.065840 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:51.066464 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:51.066486 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:51.070717 (+ 4231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:51.073672 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:51.073683 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:51.073696 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:51.074057 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:51.074445 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:51.074450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:51.074453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:51.074595 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:51.075299 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:51.075311 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:51.075501 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 15:09:51.075704 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 15:09:51.076299 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":116,"thread_start_us":305,"threads_started":1}
I20260925 15:09:51.080612 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:09:51.081620 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.081262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40375 (local address 127.0.0.1:46684)
0925 15:09:51.081419 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.081519 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 15:09:51.081881 13804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
I20260925 15:09:51.083250 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.082942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40375 (local address 127.0.0.1:46698)
0925 15:09:51.083069 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.083127 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:09:51.083415 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:40375: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.61.keytab.
Entry for principal kudu/127.22.236.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.61.keytab.
I20260925 15:09:51.098685 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.61:40375
--webserver_interface=127.22.236.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.61.keytab
--principal=kudu/127.22.236.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:42095,127.22.236.61:40375,127.22.236.60:41031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:09:51.132725 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } attempt: 1
I20260925 15:09:51.133821 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.133375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40375 (local address 127.0.0.1:46708)
0925 15:09:51.133563 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.133682 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 15:09:51.134850 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.134522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40375 (local address 127.0.0.1:46710)
0925 15:09:51.134642 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.134716 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 15:09:51.135026 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:40375: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
I20260925 15:09:51.203284 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } attempt: 2
I20260925 15:09:51.204490 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.203981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40375 (local address 127.0.0.1:46722)
0925 15:09:51.204254 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.204356 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 15:09:51.205458 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.205127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40375 (local address 127.0.0.1:46724)
0925 15:09:51.205260 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.205337 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 15:09:51.205644 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:40375: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
I20260925 15:09:51.296860 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } attempt: 3
I20260925 15:09:51.297919 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.297497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40375 (local address 127.0.0.1:46726)
0925 15:09:51.297699 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.297786 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 15:09:51.298925 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.298600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40375 (local address 127.0.0.1:46738)
0925 15:09:51.298727 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.298795 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:09:51.299115 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:40375: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
W20260925 15:09:51.408293 13861 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:09:51.408581 13861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:51.408649 13861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:51.408737 13861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:51.415926 13861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:51.416013 13861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:51.416071 13861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:51.416118 13861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:51.416206 13861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:51.425678 13861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:42095,127.22.236.61:40375,127.22.236.60:41031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.61.keytab
--principal=kudu/127.22.236.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.61:40375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.236.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:51.426862 13861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:51.428064 13861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:51.438843 13868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:51.438928 13870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:51.438867 13867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:51.439790 13861 server_base.cc:1034] running on GCE node
I20260925 15:09:51.440557 13861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:51.441838 13861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:51.442333 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } attempt: 4
I20260925 15:09:51.443073 13861 hybrid_clock.cc:648] HybridClock initialized: now 1790348991443007 us; error 29 us; skew 500 ppm
I20260925 15:09:51.443423 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.442873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40375 (local address 127.0.0.1:46754)
0925 15:09:51.443170 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.443266 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260925 15:09:51.443434 13861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:09:51.444415 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.444029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40375 (local address 127.0.0.1:46768)
0925 15:09:51.444195 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.444279 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:09:51.444612 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.61:40375: Network error: Client connection negotiation failed: client connection to 127.22.236.61:40375: connect: Connection refused (error 111)
Sep 25 15:09:51 dist-test-slave-nsrc krb5kdc[13774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348991, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:51.446630 13861 init.cc:377] Logged in from keytab as kudu/127.22.236.61@KRBTEST.COM (short username kudu)
I20260925 15:09:51.447917 13861 webserver.cc:460] Webserver started at http://127.22.236.61:41055/ using document root <none> and password file <none>
I20260925 15:09:51.448593 13861 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:51.448678 13861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:51.448928 13861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:51.451627 13861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/data/instance:
uuid: "bec2a45a93d542e38e07594f48f1ff0b"
format_stamp: "Formatted at 2026-09-25 15:09:51 on dist-test-slave-nsrc"
server_key: "761843b76c8df510583796f423c46834"
server_key_iv: "875d6df487ca08db96bb3c4ff134c81a"
server_key_version: "encryptionkey@0"
I20260925 15:09:51.452373 13861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/wal/instance:
uuid: "bec2a45a93d542e38e07594f48f1ff0b"
format_stamp: "Formatted at 2026-09-25 15:09:51 on dist-test-slave-nsrc"
server_key: "761843b76c8df510583796f423c46834"
server_key_iv: "875d6df487ca08db96bb3c4ff134c81a"
server_key_version: "encryptionkey@0"
I20260925 15:09:51.457053 13861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:09:51.460160 13877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:51.461449 13861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:51.461596 13861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/wal
uuid: "bec2a45a93d542e38e07594f48f1ff0b"
format_stamp: "Formatted at 2026-09-25 15:09:51 on dist-test-slave-nsrc"
server_key: "761843b76c8df510583796f423c46834"
server_key_iv: "875d6df487ca08db96bb3c4ff134c81a"
server_key_version: "encryptionkey@0"
I20260925 15:09:51.461790 13861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:51.486218 13861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:51.489974 13861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:51.490233 13861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:51.506474 13861 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.61:40375
I20260925 15:09:51.506496 13928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.61:40375 every 8 connection(s)
I20260925 15:09:51.507730 13861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/data/info.pb
I20260925 15:09:51.509280 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 13861
I20260925 15:09:51.509457 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-1/wal/instance
I20260925 15:09:51.509843 23472 external_mini_cluster.cc:1428] Setting key 5c32699d46a7df3a721dbcde09ee421e
I20260925 15:09:51.511456 13929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:51.517690 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:09:51 dist-test-slave-nsrc krb5kdc[13774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348991, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.61@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.236.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.236.60@KRBTEST.COM" created.
I20260925 15:09:51.537766 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.520434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.22.236.62:42095)
0925 15:09:51.520640 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:51.520646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:51.521860 (+ 1214us) server_negotiation.cc:407] Connection header received
0925 15:09:51.523693 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:51.523699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:51.523781 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:51.523888 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:51.525436 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.526337 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.527808 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.528467 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.528604 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:51.532166 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:51.532174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:51.532179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:51.532220 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:51.534700 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:51.535557 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:51.535561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:51.535564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:51.535680 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:51.536136 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:51.536161 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:51.536165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:51.536455 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:51.536564 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:51.537349 (+ 785us) server_negotiation.cc:299] Negotiation successful
0925 15:09:51.537527 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:09:51.539086 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.520919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:42095 (local address 127.0.0.1:53902)
0925 15:09:51.521692 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.521740 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:51.523445 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:51.524075 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:51.524082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:51.524647 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:51.525266 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:51.525284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.526484 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.526487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:51.527672 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:51.527683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.528674 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.528679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:51.528966 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:51.529635 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:51.529657 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:51.531900 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:51.534848 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:51.534855 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:51.534867 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:51.535448 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:51.535799 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:51.535804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:51.535807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:51.536036 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:51.536571 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:51.536580 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:51.537007 (+ 427us) client_negotiation.cc:769] Sending connection context
0925 15:09:51.537306 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 15:09:51.537812 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":305,"threads_started":1}
I20260925 15:09:51.540715 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:09:51 dist-test-slave-nsrc krb5kdc[13774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348991, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.61@KRBTEST.COM for kudu/127.22.236.61@KRBTEST.COM
I20260925 15:09:51.554855 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.541392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40375 (local address 127.0.0.1:46770)
0925 15:09:51.541590 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.541607 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:51.541718 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:51.542665 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:51.542669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:51.543102 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:51.543280 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:51.543291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.544642 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.544647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:51.545910 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:51.545923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.546670 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.546679 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:51.546833 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:51.547328 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:51.547348 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:51.549620 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:51.552450 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:51.552455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:51.552459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:51.552934 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:51.553243 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:51.553248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:51.553252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:51.553412 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:51.553930 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:51.553935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:51.554069 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 15:09:51.554205 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 15:09:51.554471 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 15:09:51.554872 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.541648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.22.236.61:40375)
0925 15:09:51.542268 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:51.542274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:51.542297 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:09:51.542338 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:51.542343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:51.542510 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:51.542614 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:51.543440 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.544507 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.546028 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.546517 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.546608 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:51.549800 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:51.549811 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:51.549823 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:51.549853 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:51.552344 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:51.553041 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:51.553047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:51.553049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:51.553156 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:51.553511 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:51.553515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:51.553517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:51.553811 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:51.553974 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:51.554304 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 15:09:51.554593 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":486,"thread_start_us":354,"threads_started":1}
I20260925 15:09:51.557236 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:09:51.558270 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.557771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35068)
0925 15:09:51.557957 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.558104 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 15:09:51.558569 13878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
I20260925 15:09:51.559823 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.559558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35078)
0925 15:09:51.559661 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.559734 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:09:51.560014 13929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:41031: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.236.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.60.keytab.
Entry for principal kudu/127.22.236.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.60.keytab.
I20260925 15:09:51.567046 23472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSyRird/build/asan/bin/kudu
/tmp/dist-test-taskSyRird/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.60:41031
--webserver_interface=127.22.236.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.60.keytab
--principal=kudu/127.22.236.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.236.20:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.236.62:42095,127.22.236.61:40375,127.22.236.60:41031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:09:51.613013 13929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } attempt: 1
I20260925 15:09:51.614110 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.613672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35092)
0925 15:09:51.613848 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.613960 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:09:51.615088 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.614747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35106)
0925 15:09:51.614907 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.614980 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:09:51.615350 13929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:41031: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
I20260925 15:09:51.683559 13929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } attempt: 2
I20260925 15:09:51.684742 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.684204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35118)
0925 15:09:51.684433 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.684540 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:09:51.685736 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.685415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35132)
0925 15:09:51.685550 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.685621 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:09:51.685992 13929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:41031: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
I20260925 15:09:51.743857 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } attempt: 5
Sep 25 15:09:51 dist-test-slave-nsrc krb5kdc[13774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348990, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for kudu/127.22.236.61@KRBTEST.COM
I20260925 15:09:51.756191 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.744507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40375 (local address 127.0.0.1:46786)
0925 15:09:51.744812 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.744832 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:51.744974 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:51.745293 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:51.745298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:51.745659 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:51.745838 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:51.745846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.746864 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.746868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:51.747709 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:51.747720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.748523 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.748529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:51.748704 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:51.749227 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:51.749245 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:51.751095 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:51.753928 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:51.753934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:51.753938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:51.754274 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:51.754614 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:51.754619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:51.754623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:51.754769 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:51.755257 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:51.755261 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:51.755405 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 15:09:51.755524 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 15:09:51.755816 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260925 15:09:51.756202 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.744688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.22.236.61:40375)
0925 15:09:51.744878 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:51.744883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:51.744903 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:09:51.745095 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:51.745101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:51.745162 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:51.745242 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:51.745982 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.746734 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.747846 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:51.748379 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:51.748497 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:51.751263 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:51.751268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:51.751272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:51.751304 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:51.753798 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:51.754376 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:51.754382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:51.754384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:51.754513 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:51.754864 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:51.754869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:51.754872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:51.755149 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:51.755272 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:51.755645 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 15:09:51.755836 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:09:51.757894 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:09:51.758836 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.758496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35138)
0925 15:09:51.758644 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.758713 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:09:51.759626 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.759335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35140)
0925 15:09:51.759460 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.759536 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 15:09:51.759848 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:41031: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
I20260925 15:09:51.777177 13929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } attempt: 3
I20260925 15:09:51.778124 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.777714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35144)
0925 15:09:51.777894 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.777978 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:09:51.779095 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.778702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35158)
0925 15:09:51.778919 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.778990 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260925 15:09:51.779330 13929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:41031: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
I20260925 15:09:51.811044 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } attempt: 1
I20260925 15:09:51.812071 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.811637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35160)
0925 15:09:51.811811 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.811913 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 15:09:51.813037 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.812718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35172)
0925 15:09:51.812850 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.812927 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:09:51.813269 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:41031: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
I20260925 15:09:51.881466 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } attempt: 2
I20260925 15:09:51.882527 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.882074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35182)
0925 15:09:51.882291 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.882379 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 15:09:51.883530 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.883224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35184)
0925 15:09:51.883347 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.883426 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:09:51.883713 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:41031: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
I20260925 15:09:51.922509 13929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } attempt: 4
I20260925 15:09:51.923512 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.923078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35194)
0925 15:09:51.923287 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.923371 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 15:09:51.924398 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.924090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35208)
0925 15:09:51.924240 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.924302 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:09:51.924614 13929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:41031: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
W20260925 15:09:51.977912 13936 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:09:51.978222 13936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:51.978322 13936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:51.978431 13936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:51.985519 13936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:51.985606 13936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:51.985698 13936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:51.985769 13936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:51.985852 13936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:51.989900 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } attempt: 3
I20260925 15:09:51.990933 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.990515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35222)
0925 15:09:51.990689 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.990775 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 15:09:51.991833 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:51.991513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35226)
0925 15:09:51.991669 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:51.991734 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260925 15:09:51.992002 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.236.60:41031: Network error: Client connection negotiation failed: client connection to 127.22.236.60:41031: connect: Connection refused (error 111)
I20260925 15:09:51.995960 13936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.236.20:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.236.62:42095,127.22.236.61:40375,127.22.236.60:41031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/krb5kdc/kudu_127.22.236.60.keytab
--principal=kudu/127.22.236.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.236.60:41031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.236.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:09:51.997167 13936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:51.998368 13936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:52.009354 13942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:52.009374 13941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:52.009474 13944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:52.009950 13936 server_base.cc:1034] running on GCE node
I20260925 15:09:52.010735 13936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:52.012173 13936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:52.013379 13936 hybrid_clock.cc:648] HybridClock initialized: now 1790348992013331 us; error 39 us; skew 500 ppm
I20260925 15:09:52.013677 13936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:52 dist-test-slave-nsrc krb5kdc[13774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348992, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:52.016594 13936 init.cc:377] Logged in from keytab as kudu/127.22.236.60@KRBTEST.COM (short username kudu)
I20260925 15:09:52.017867 13936 webserver.cc:460] Webserver started at http://127.22.236.60:40663/ using document root <none> and password file <none>
I20260925 15:09:52.018491 13936 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:52.018577 13936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:52.018862 13936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:09:52.021754 13936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/data/instance:
uuid: "784e48b9c3c24115b28de75d9c780eb9"
format_stamp: "Formatted at 2026-09-25 15:09:52 on dist-test-slave-nsrc"
server_key: "b2c75a0e977db85419243becdfd0e7ec"
server_key_iv: "70bb5cb0a573e23690abe95b45c4f0f8"
server_key_version: "encryptionkey@0"
I20260925 15:09:52.022475 13936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/wal/instance:
uuid: "784e48b9c3c24115b28de75d9c780eb9"
format_stamp: "Formatted at 2026-09-25 15:09:52 on dist-test-slave-nsrc"
server_key: "b2c75a0e977db85419243becdfd0e7ec"
server_key_iv: "70bb5cb0a573e23690abe95b45c4f0f8"
server_key_version: "encryptionkey@0"
I20260925 15:09:52.027148 13936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:09:52.030285 13951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:52.031464 13936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:09:52.031600 13936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/data,/tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/wal
uuid: "784e48b9c3c24115b28de75d9c780eb9"
format_stamp: "Formatted at 2026-09-25 15:09:52 on dist-test-slave-nsrc"
server_key: "b2c75a0e977db85419243becdfd0e7ec"
server_key_iv: "70bb5cb0a573e23690abe95b45c4f0f8"
server_key_version: "encryptionkey@0"
I20260925 15:09:52.031788 13936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:52.052850 13936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:52.056219 13936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:52.056468 13936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:52.072026 13936 rpc_server.cc:307] RPC server started. Bound to: 127.22.236.60:41031
I20260925 15:09:52.072048 14002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.236.60:41031 every 8 connection(s)
I20260925 15:09:52.073251 13936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/data/info.pb
I20260925 15:09:52.076718 14003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:52.076928 23472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSyRird/build/asan/bin/kudu as pid 13936
I20260925 15:09:52.077073 23472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSyRird/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348645297121-23472-0/minicluster-data/master-2/wal/instance
I20260925 15:09:52.077476 23472 external_mini_cluster.cc:1428] Setting key 98ed7024bd57927e330e11c6f5facdc6
I20260925 15:09:52.086457 14003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:09:52 dist-test-slave-nsrc krb5kdc[13774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.60@KRBTEST.COM
Sep 25 15:09:52 dist-test-slave-nsrc krb5kdc[13774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348992, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.60@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
I20260925 15:09:52.102710 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.079012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.22.236.60:41031)
0925 15:09:52.079742 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.079756 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.079796 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:09:52.081253 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.081271 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.081650 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.081961 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.083556 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.085370 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.087265 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.088396 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.088564 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.094154 (+ 5590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:52.094164 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:52.094176 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:52.094218 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:52.097435 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.098266 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.098271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.098294 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.098601 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.098957 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.098962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.098966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.099577 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:52.099839 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.101007 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.101462 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"spinlock_wait_cycles":30848,"thread_start_us":348,"threads_started":1}
I20260925 15:09:52.107487 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.089598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.62:42095 (local address 127.0.0.1:53910)
0925 15:09:52.091408 (+ 1810us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:52.091459 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:52.091612 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:52.092016 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:52.092023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:52.092835 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:52.093161 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:52.093174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.094680 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.094686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:52.096385 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:52.096406 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.097458 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.097466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:52.097698 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.098332 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:52.098354 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:52.101025 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:52.104497 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:52.104507 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:52.104514 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:52.105115 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:52.105555 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:52.105560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:52.105563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:52.105705 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:52.106445 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:52.106453 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:52.106632 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 15:09:52.106796 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 15:09:52.107100 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1567,"thread_start_us":1424,"threads_started":1}
I20260925 15:09:52.108489 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.089603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.22.236.62:42095)
0925 15:09:52.090174 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.090181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.091552 (+ 1371us) server_negotiation.cc:407] Connection header received
0925 15:09:52.091742 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.091747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.091827 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.091940 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.093347 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.094507 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.096574 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.097209 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.097340 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.101268 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:52.101273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:52.101277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:52.101309 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:52.104365 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.105264 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.105269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.105273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.105425 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.105830 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.105835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.105838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.106204 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:52.106361 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.107778 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.108032 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":288,"threads_started":1}
I20260925 15:09:52.109921 14003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:09:52 dist-test-slave-nsrc krb5kdc[13774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.62@KRBTEST.COM
Sep 25 15:09:52 dist-test-slave-nsrc krb5kdc[13774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348992, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.60@KRBTEST.COM for kudu/127.22.236.61@KRBTEST.COM
I20260925 15:09:52.120522 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.105979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.22.236.62:42095)
0925 15:09:52.106522 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.106526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.106544 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:09:52.106580 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.106583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.106634 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.106723 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.108123 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.109327 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.110841 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.111392 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.111511 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.115436 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:52.115443 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:52.115446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:52.115479 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:52.118196 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.118861 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.118864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.118866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.118976 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.119316 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.119320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.119322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.119573 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:52.119756 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.120064 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.120251 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":247,"threads_started":1}
I20260925 15:09:52.122880 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.110514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.61:40375 (local address 127.0.0.1:46794)
0925 15:09:52.110673 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:52.110690 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:52.110779 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:52.111513 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:52.111517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:52.111843 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:52.111978 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:52.111985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.112995 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.113000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:52.113953 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:52.113966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.114669 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.114675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:52.114851 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.115295 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:52.115315 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:52.117371 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:52.120736 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:52.120741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:52.120745 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:52.121174 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:52.121544 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:52.121549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:52.121552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:52.121706 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:52.122249 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:52.122253 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:52.122389 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 15:09:52.122497 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:09:52.122673 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:09:52.123234 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.110902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.22.236.61:40375)
0925 15:09:52.111092 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.111098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.111115 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:09:52.111170 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.111175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.111248 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.111334 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.112128 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.112875 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.114077 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.114520 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.114624 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.117539 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:52.117544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:52.117547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:52.117578 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:52.120494 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.121281 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.121286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.121290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.121455 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.121804 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.121809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.121813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.122128 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:52.122273 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.122603 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.122818 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:09:52.123974 14003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:09:52 dist-test-slave-nsrc krb5kdc[13774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348992, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.60@KRBTEST.COM for kudu/127.22.236.60@KRBTEST.COM
Sep 25 15:09:52 dist-test-slave-nsrc krb5kdc[13774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.236.61@KRBTEST.COM
I20260925 15:09:52.135712 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.124578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35244)
0925 15:09:52.124724 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:52.124740 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:52.124830 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:52.125080 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:52.125084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:52.125414 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:52.125526 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:52.125533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.126382 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.126387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:52.127198 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:52.127208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.127827 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.127832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:52.127994 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.128477 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:52.128490 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:52.130474 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:52.133580 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:52.133583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:52.133585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:52.133907 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:52.134269 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:52.134272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:52.134274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:52.134402 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:52.134857 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:52.134861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:52.135038 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 15:09:52.135176 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 15:09:52.135450 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:09:52.136027 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.124611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.22.236.60:41031)
0925 15:09:52.124745 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.124750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.124783 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:09:52.124930 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.124935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.124989 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.125056 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.125639 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.126277 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.127311 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.127694 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.127792 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.130618 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:52.130625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:52.130629 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:52.130660 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:52.133468 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.134028 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.134033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.134036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.134169 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.134502 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.134506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.134508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.134761 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:52.134861 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.135262 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.135629 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":49}
I20260925 15:09:52.137066 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.122024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.22.236.61:40375)
0925 15:09:52.123540 (+ 1516us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.123545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.123565 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:09:52.123604 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.123609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.123662 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.123747 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.125087 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.125828 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.126977 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.127363 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.127464 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.131408 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:52.131414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:52.131419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:52.131454 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:52.134310 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.134960 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.134964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.134966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.135090 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.135460 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.135465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.135468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.135770 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:52.136031 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.136524 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.136721 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1368,"thread_start_us":1177,"threads_started":1}
I20260925 15:09:52.141145 14003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9: Bootstrap starting.
I20260925 15:09:52.141250 23472 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:09:52.144649 14003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:52.145623 14003 log.cc:824] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:52.144004 13829 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1a436229ec8243879f31021f400d2f83"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53918
I20260925 15:09:52.148243 14003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9: No bootstrap required, opened a new log
I20260925 15:09:52.154914 14003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } }
I20260925 15:09:52.155215 14003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 784e48b9c3c24115b28de75d9c780eb9, State: Initialized, Role: FOLLOWER
I20260925 15:09:52.155628 14003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } }
I20260925 15:09:52.156482 14012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } } }
I20260925 15:09:52.156848 14012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:09:52.157426 14003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:09:52.166014 14023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:09:52.166132 14023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:09:52.169215 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } attempt: 4
I20260925 15:09:52.163810 13936 master_runner.cc:186] Error getting master registration for 127.22.236.62:42095: OK, instance_id { permanent_uuid: "1a436229ec8243879f31021f400d2f83" instance_seqno: 1790348991037527 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:09:52.173874 13936 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 15:09:52 dist-test-slave-nsrc krb5kdc[13774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348990, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.62@KRBTEST.COM for kudu/127.22.236.60@KRBTEST.COM
I20260925 15:09:52.184546 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.169869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35250)
0925 15:09:52.170125 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:52.170144 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:52.170250 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:52.170792 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:52.170799 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:52.171303 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:52.171518 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:52.171531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.172952 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.172958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:52.174300 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:52.174316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.175045 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.175052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:52.175283 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.175880 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:52.175910 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:52.178007 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:52.181895 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:52.181901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:52.181905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:52.182376 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:52.182775 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:52.182779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:52.182783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:52.182925 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:52.183539 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:52.183543 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:52.183716 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 15:09:52.183851 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 15:09:52.184112 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260925 15:09:52.184592 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.170092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.22.236.60:41031)
0925 15:09:52.170467 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.170472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.170488 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:09:52.170536 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.170540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.170617 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.170702 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.171677 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.172796 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.174445 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.174894 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.175017 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.178201 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:52.178207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:52.178213 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:52.178249 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:52.181756 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.182505 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.182510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.182514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.182671 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.183028 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.183033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.183037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.183430 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:52.183595 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.183973 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.184225 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260925 15:09:52.190459 13854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83: Bootstrap starting.
I20260925 15:09:52.194727 13854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:52.196089 13854 log.cc:824] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:52.199503 13854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83: No bootstrap required, opened a new log
I20260925 15:09:52.207015 13854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } }
I20260925 15:09:52.207425 13854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a436229ec8243879f31021f400d2f83, State: Initialized, Role: FOLLOWER
I20260925 15:09:52.207952 13854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } }
I20260925 15:09:52.208925 14026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } } }
I20260925 15:09:52.209295 14026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:09:52.210192 13854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:52.210263 13829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:09:52.210439 13829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:52.212922 13829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } }
I20260925 15:09:52.214057 13829 leader_election.cc:288] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [CANDIDATE]: Term 1 election: Requested vote from peers bec2a45a93d542e38e07594f48f1ff0b (127.22.236.61:40375), 784e48b9c3c24115b28de75d9c780eb9 (127.22.236.60:41031)
I20260925 15:09:52.215520 13978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a436229ec8243879f31021f400d2f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "784e48b9c3c24115b28de75d9c780eb9"
I20260925 15:09:52.216008 13978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:52.219455 13978 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a436229ec8243879f31021f400d2f83 in term 1.
I20260925 15:09:52.220175 13803 leader_election.cc:302] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a436229ec8243879f31021f400d2f83, 784e48b9c3c24115b28de75d9c780eb9; no voters:
I20260925 15:09:52.220597 14026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:52.221271 14026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [term 1 LEADER]: Becoming Leader. State: Replica: 1a436229ec8243879f31021f400d2f83, State: Running, Role: LEADER
I20260925 15:09:52.221817 14026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } }
I20260925 15:09:52.223752 13929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } attempt: 5
Sep 25 15:09:52 dist-test-slave-nsrc krb5kdc[13774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348991, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.236.61@KRBTEST.COM for kudu/127.22.236.60@KRBTEST.COM
I20260925 15:09:52.246151 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.224364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.236.60:41031 (local address 127.0.0.1:35266)
0925 15:09:52.224746 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:52.224766 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:52.224878 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:52.225782 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:52.225787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:52.226288 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:52.226492 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:52.226503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.228021 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.228027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:52.229436 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:52.229451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.230678 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.230684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:52.230927 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.231608 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:52.231654 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:52.235650 (+ 3996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:52.240728 (+ 5078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:52.240732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:52.240736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:52.241528 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:52.242166 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:52.242170 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:52.242194 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:52.242412 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:52.243471 (+ 1059us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:52.243476 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:52.243726 (+ 250us) client_negotiation.cc:769] Sending connection context
0925 15:09:52.243929 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 15:09:52.244835 (+ 906us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260925 15:09:52.247258 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.225095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.22.236.60:41031)
0925 15:09:52.225264 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.225269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.225285 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:09:52.225341 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.225346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.225467 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.225555 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.226690 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.227830 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.229653 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.230345 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.230460 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.235909 (+ 5449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:52.235916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:52.235920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:52.235957 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:52.240529 (+ 4572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.241687 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.241692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.241695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.242014 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.242615 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.242620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.242651 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.243265 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:52.244365 (+ 1100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.244887 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.245682 (+ 795us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:09:52.253444 13929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b: Bootstrap starting.
I20260925 15:09:52.264591 13929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:52.266567 14032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a436229ec8243879f31021f400d2f83. Latest consensus state: current_term: 1 leader_uuid: "1a436229ec8243879f31021f400d2f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } } }
I20260925 15:09:52.271073 14032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:52.273691 13929 log.cc:824] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b: Log is configured to *not* fsync() on all Append() calls
W20260925 15:09:52.278124 13808 tablet.cc:2369] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:09:52.278771 14046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:52.281286 14048 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:09:52.302219 13929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b: No bootstrap required, opened a new log
I20260925 15:09:52.285543 13787 master_runner.cc:186] Error getting master registration for 127.22.236.61:40375: OK, instance_id { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" instance_seqno: 1790348991501779 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260925 15:09:52.308318 13957 tablet.cc:2369] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:09:52.308420 13787 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:09:52.312309 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.259063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.22.236.61:40375)
0925 15:09:52.259249 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.259255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.269171 (+ 9916us) server_negotiation.cc:407] Connection header received
0925 15:09:52.270101 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.270107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.270218 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.270301 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.280837 (+ 10536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.281971 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.289409 (+ 7438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.290085 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.290189 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.300966 (+ 10777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:52.300974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:52.300979 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:52.301021 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:52.305483 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.306293 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.306299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.306303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.306459 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.310360 (+ 3901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.310367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.310370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.310774 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:52.311012 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.311702 (+ 690us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.311928 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:09:52.315773 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.269476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.22.236.60:41031)
0925 15:09:52.270304 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.270310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.273697 (+ 3387us) server_negotiation.cc:407] Connection header received
0925 15:09:52.276224 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.276230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.276308 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.276373 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.286517 (+ 10144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.287605 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.294996 (+ 7391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.295705 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.295806 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.301895 (+ 6089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:52.301902 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:52.301906 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:52.301949 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:52.306262 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.308176 (+ 1914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.308183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.308187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.308337 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.312338 (+ 4001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.312344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.312347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.313455 (+ 1108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:52.313691 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.315078 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.315382 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564}
I20260925 15:09:52.319200 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.261562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.22.236.62:42095)
0925 15:09:52.261746 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.261752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.273062 (+ 11310us) server_negotiation.cc:407] Connection header received
0925 15:09:52.273940 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.273946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.274047 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.274119 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.283695 (+ 9576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.284776 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.291320 (+ 6544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.292129 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.292274 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.303134 (+ 10860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:52.303146 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:52.303151 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:52.303195 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:52.310021 (+ 6826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.311913 (+ 1892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.311919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.311922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.313878 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.314590 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.314595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.314599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.315680 (+ 1081us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:52.315895 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.318536 (+ 2641us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.318778 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:09:52.321489 13929 raft_consensus.cc:348] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } }
I20260925 15:09:52.321899 13929 raft_consensus.cc:729] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bec2a45a93d542e38e07594f48f1ff0b, State: Initialized, Role: FOLLOWER
I20260925 15:09:52.322404 13929 consensus_queue.cc:260] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } }
I20260925 15:09:52.326802 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } } }
I20260925 15:09:52.327502 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:09:52.327994 13929 sys_catalog.cc:564] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:52.319211 14046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:52.328037 13904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a436229ec8243879f31021f400d2f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bec2a45a93d542e38e07594f48f1ff0b"
I20260925 15:09:52.328529 13904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:52.331840 13904 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a436229ec8243879f31021f400d2f83 in term 1.
I20260925 15:09:52.337040 13903 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b [term 1 FOLLOWER]: Refusing update from remote peer 1a436229ec8243879f31021f400d2f83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:52.338176 13978 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9 [term 1 FOLLOWER]: Refusing update from remote peer 1a436229ec8243879f31021f400d2f83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:52.339056 14032 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:52.340487 14032 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:52.371236 14012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a436229ec8243879f31021f400d2f83. Latest consensus state: current_term: 1 leader_uuid: "1a436229ec8243879f31021f400d2f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } } }
I20260925 15:09:52.371554 14012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:09:52.380614 14032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a436229ec8243879f31021f400d2f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } } }
I20260925 15:09:52.381043 14032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:52.383337 14046 catalog_manager.cc:1347] Generated new cluster ID: 131b2d107d414a058092669050bbafdb
I20260925 15:09:52.383455 14046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:52.395385 14012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a436229ec8243879f31021f400d2f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } } }
I20260925 15:09:52.395644 14012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:09:52.401257 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a436229ec8243879f31021f400d2f83. Latest consensus state: current_term: 1 leader_uuid: "1a436229ec8243879f31021f400d2f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } } }
I20260925 15:09:52.401634 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:09:52.406262 14032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a436229ec8243879f31021f400d2f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } } }
I20260925 15:09:52.406517 14032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:52.412632 14054 mvcc.cc:205] Tried to move back new op lower bound from 7333269472600539136 to 7333269472145350656. Current Snapshot: MvccSnapshot[applied={T|T < 7333269472600539136}]
I20260925 15:09:52.423870 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a436229ec8243879f31021f400d2f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a436229ec8243879f31021f400d2f83" member_type: VOTER last_known_addr { host: "127.22.236.62" port: 42095 } } peers { permanent_uuid: "bec2a45a93d542e38e07594f48f1ff0b" member_type: VOTER last_known_addr { host: "127.22.236.61" port: 40375 } } peers { permanent_uuid: "784e48b9c3c24115b28de75d9c780eb9" member_type: VOTER last_known_addr { host: "127.22.236.60" port: 41031 } } }
I20260925 15:09:52.424495 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:09:52.439962 14046 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:52.441749 14046 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:52.460325 14046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a436229ec8243879f31021f400d2f83: Generated new TSK 0
I20260925 15:09:52.461377 14046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:09:52.471174 14069 catalog_manager.cc:1259] Loaded cluster ID: 131b2d107d414a058092669050bbafdb
I20260925 15:09:52.471357 14069 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b: loading cluster ID for follower catalog manager: success
I20260925 15:09:52.476058 14069 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b: acquiring CA information for follower catalog manager: success
I20260925 15:09:52.478488 14069 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bec2a45a93d542e38e07594f48f1ff0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 15:09:52.601495 14009 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:52.587233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.22.236.62:42095)
0925 15:09:52.587516 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.587523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.598444 (+ 10921us) server_negotiation.cc:407] Connection header received
0925 15:09:52.599829 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.599836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.599929 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.600003 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.601197 (+ 1194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53932: BlockingRecv error: recv got EOF from 127.0.0.1:53932 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260925 15:09:52.604979 14010 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:52.594153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.22.236.61:40375)
0925 15:09:52.594334 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.594341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.602893 (+ 8552us) server_negotiation.cc:407] Connection header received
0925 15:09:52.603558 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.603565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.603671 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.603758 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.604735 (+ 977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46822: BlockingRecv error: recv got EOF from 127.0.0.1:46822 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 15:09:52.606750 14006 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:52.598994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.22.236.60:41031)
0925 15:09:52.599162 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.599168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.604595 (+ 5427us) server_negotiation.cc:407] Connection header received
0925 15:09:52.605037 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.605043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.605137 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.605205 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.606530 (+ 1325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35276: BlockingRecv error: recv got EOF from 127.0.0.1:35276 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:09:52.666821 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.647642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.22.236.61:40375)
0925 15:09:52.647830 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.647837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.658399 (+ 10562us) server_negotiation.cc:407] Connection header received
0925 15:09:52.658939 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.658945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.659165 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.659303 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:52.660128 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.661257 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.663641 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.664354 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.664444 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.665237 (+ 793us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:52.665982 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:52.666097 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.666381 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.666481 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 15:09:52.710531 14006 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:52.699819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.22.236.60:41031)
0925 15:09:52.700020 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.700026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.708739 (+ 8713us) server_negotiation.cc:407] Connection header received
0925 15:09:52.708978 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.708985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.709113 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.709211 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.710272 (+ 1061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35284: BlockingRecv error: recv got EOF from 127.0.0.1:35284 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:09:52.750900 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.734126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.22.236.61:40375)
0925 15:09:52.734336 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.734343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.744548 (+ 10205us) server_negotiation.cc:407] Connection header received
0925 15:09:52.744784 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.744790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.744886 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.744990 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:52.745558 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.746812 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.748772 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.749462 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.749592 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.750079 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:52.750233 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:52.750326 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.750482 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.750566 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260925 15:09:52.762168 14006 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:52.758347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.22.236.60:41031)
0925 15:09:52.758567 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.758573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.760683 (+ 2110us) server_negotiation.cc:407] Connection header received
0925 15:09:52.760880 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.760887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.760986 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.761097 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.761943 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35300: BlockingRecv error: recv got EOF from 127.0.0.1:35300 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:09:52.778862 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.769864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.22.236.61:40375)
0925 15:09:52.770054 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.770060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.772660 (+ 2600us) server_negotiation.cc:407] Connection header received
0925 15:09:52.772842 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.772852 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.772944 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.773039 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:52.773445 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.774595 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.776609 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.777296 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.777398 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.778032 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:52.778202 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:52.778299 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.778465 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.778548 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 15:09:52.787473 14006 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:52.785630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.22.236.60:41031)
0925 15:09:52.785808 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.785814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.785833 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:09:52.786038 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.786045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.786148 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.786221 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.787232 (+ 1011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35310: BlockingRecv error: recv got EOF from 127.0.0.1:35310 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:09:52.801939 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.795207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.22.236.61:40375)
0925 15:09:52.795435 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.795440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.795982 (+ 542us) server_negotiation.cc:407] Connection header received
0925 15:09:52.796176 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.796182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.796270 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.796380 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:52.796835 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.797974 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.799868 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.800549 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.800679 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.801137 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:52.801296 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:52.801399 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.801558 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.801639 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260925 15:09:52.810612 14006 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:52.808457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35318 (local address 127.22.236.60:41031)
0925 15:09:52.808699 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.808706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.809203 (+ 497us) server_negotiation.cc:407] Connection header received
0925 15:09:52.809368 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.809374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.809478 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.809582 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.810383 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35318: BlockingRecv error: recv got EOF from 127.0.0.1:35318 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
I20260925 15:09:52.828791 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.821984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.22.236.61:40375)
0925 15:09:52.822164 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.822170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.822659 (+ 489us) server_negotiation.cc:407] Connection header received
0925 15:09:52.822855 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.822861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.822971 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.823066 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:52.823508 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.824723 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.826629 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.827384 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.827511 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.827957 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:52.828089 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:52.828245 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.828377 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.828477 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 15:09:52.837688 14006 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:52.835773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.22.236.60:41031)
0925 15:09:52.835948 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.835954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.836260 (+ 306us) server_negotiation.cc:407] Connection header received
0925 15:09:52.836456 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.836462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.836559 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.836666 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.837457 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35320: BlockingRecv error: recv got EOF from 127.0.0.1:35320 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:09:52.864758 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.857815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46866 (local address 127.22.236.61:40375)
0925 15:09:52.857996 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.858002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.858505 (+ 503us) server_negotiation.cc:407] Connection header received
0925 15:09:52.858741 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.858748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.858847 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.858954 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:52.859452 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.860710 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.862647 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.863295 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.863429 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.863902 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:52.864038 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:52.864130 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.864316 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.864399 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 15:09:52.876304 14006 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:52.874069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.22.236.60:41031)
0925 15:09:52.874244 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.874250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.874788 (+ 538us) server_negotiation.cc:407] Connection header received
0925 15:09:52.874960 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.874966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.875067 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.875178 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.876031 (+ 853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35336: BlockingRecv error: recv got EOF from 127.0.0.1:35336 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:09:52.917120 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.911279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46882 (local address 127.22.236.61:40375)
0925 15:09:52.911505 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.911512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.911843 (+ 331us) server_negotiation.cc:407] Connection header received
0925 15:09:52.912001 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.912007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.912104 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.912300 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:52.912749 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.913950 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.915264 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.915881 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.916020 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.916366 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:52.916534 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:52.916615 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.916715 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.916795 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 15:09:52.924521 14006 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:52.922746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.22.236.60:41031)
0925 15:09:52.922930 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.922936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.923202 (+ 266us) server_negotiation.cc:407] Connection header received
0925 15:09:52.923350 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.923356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.923453 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.923550 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.924304 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35344: BlockingRecv error: recv got EOF from 127.0.0.1:35344 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:09:52.996526 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.992001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.22.236.61:40375)
0925 15:09:52.992183 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.992189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.992509 (+ 320us) server_negotiation.cc:407] Connection header received
0925 15:09:52.992691 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.992695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.992765 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.992835 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:52.993171 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.993831 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.995107 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.995507 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.995598 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.995945 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:52.996067 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:52.996133 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.996257 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.996319 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 15:09:53.003412 14006 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:53.001855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.22.236.60:41031)
0925 15:09:53.002058 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:53.002064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:53.002306 (+ 242us) server_negotiation.cc:407] Connection header received
0925 15:09:53.002441 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:53.002444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:53.002514 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:53.002589 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:53.003207 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35360: BlockingRecv error: recv got EOF from 127.0.0.1:35360 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:09:53.139374 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:53.134692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.22.236.61:40375)
0925 15:09:53.134843 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:53.134847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:53.135230 (+ 383us) server_negotiation.cc:407] Connection header received
0925 15:09:53.135410 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:53.135414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:53.135485 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:53.135566 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:53.135949 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:53.136754 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:53.138011 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:53.138414 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:53.138506 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:53.138832 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:53.138928 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:53.138987 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:53.139084 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 15:09:53.139144 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 15:09:53.146080 14006 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:09:53.144292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.22.236.60:41031)
0925 15:09:53.144487 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:53.144493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:53.144812 (+ 319us) server_negotiation.cc:407] Connection header received
0925 15:09:53.144984 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:53.144990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:53.145065 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:53.145144 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:53.145849 (+ 705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35366: BlockingRecv error: recv got EOF from 127.0.0.1:35366 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:09:53.168032 14023 catalog_manager.cc:1259] Loaded cluster ID: 131b2d107d414a058092669050bbafdb
I20260925 15:09:53.168128 14023 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9: loading cluster ID for follower catalog manager: success
I20260925 15:09:53.170632 14023 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9: acquiring CA information for follower catalog manager: success
I20260925 15:09:53.172547 14023 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 784e48b9c3c24115b28de75d9c780eb9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:09:53.410418 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:53.405708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46922 (local address 127.22.236.61:40375)
0925 15:09:53.405934 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:53.405940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:53.406121 (+ 181us) server_negotiation.cc:407] Connection header received
0925 15:09:53.406328 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:53.406332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:53.406396 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:53.406472 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:53.406884 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:53.407567 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:53.408885 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:53.409276 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:53.409381 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:53.409765 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:53.409863 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:53.409943 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:53.410071 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 15:09:53.410145 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:09:53.422569 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:53.417710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.22.236.60:41031)
0925 15:09:53.417860 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:53.417865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:53.418128 (+ 263us) server_negotiation.cc:407] Connection header received
0925 15:09:53.418263 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:53.418266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:53.418419 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:53.418512 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:53.418817 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:53.419494 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:53.420737 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:53.421105 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:53.421188 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:53.421577 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:53.422095 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:53.422195 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:53.422282 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 15:09:53.422342 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:09:53.437705 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:53.427690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.22.236.62:42095)
0925 15:09:53.431146 (+ 3456us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:53.431150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:53.431164 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:09:53.431251 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:53.431255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:53.431400 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:53.431531 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:53.432162 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:53.433364 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:53.435573 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:53.436190 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:53.436302 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:53.436740 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:53.437062 (+ 322us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:53.437158 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:53.437295 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:09:53.437376 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3364,"thread_start_us":3264,"threads_started":1}
I20260925 15:09:53.440239 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:53.427721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.22.236.61:40375)
0925 15:09:53.427920 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:53.427926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:53.434088 (+ 6162us) server_negotiation.cc:407] Connection header received
0925 15:09:53.434343 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:53.434349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:53.434444 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:53.434547 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:53.435212 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:53.436085 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:53.438415 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:53.439027 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:53.439149 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:53.439585 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:53.439685 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:53.439756 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:53.439888 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 15:09:53.439971 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:09:53.440789 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:53.428996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.22.236.60:41031)
0925 15:09:53.429170 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:53.429176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:53.434364 (+ 5188us) server_negotiation.cc:407] Connection header received
0925 15:09:53.434507 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:53.434511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:53.434598 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:53.434676 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:53.435381 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:53.436411 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:53.438812 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:53.439322 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:53.439435 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:53.440106 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:53.440245 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:53.440326 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:53.440485 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:09:53.440558 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:09:53.446381 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 13787
I20260925 15:09:53.447245 13849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:53.555171 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 13787
I20260925 15:09:53.574087 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 13861
W20260925 15:09:53.574143 13880 connection.cc:535] server connection from 127.0.0.1:46786 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:09:53.575234 13924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:53.687196 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 13861
I20260925 15:09:53.703516 23472 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSyRird/build/asan/bin/kudu pid 13936
I20260925 15:09:53.704567 13998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:53.808652 23472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSyRird/build/asan/bin/kudu with pid 13936
2026-09-25T15:09:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4430 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9021 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (348477 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST